jmo-security 0.3.2__tar.gz → 0.4.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of jmo-security might be problematic. Click here for more details.

Files changed (50) hide show
  1. {jmo_security-0.3.2 → jmo_security-0.4.1}/PKG-INFO +263 -149
  2. {jmo_security-0.3.2 → jmo_security-0.4.1}/README.md +262 -148
  3. {jmo_security-0.3.2 → jmo_security-0.4.1}/jmo_security.egg-info/PKG-INFO +263 -149
  4. {jmo_security-0.3.2 → jmo_security-0.4.1}/jmo_security.egg-info/SOURCES.txt +1 -0
  5. {jmo_security-0.3.2 → jmo_security-0.4.1}/pyproject.toml +25 -1
  6. jmo_security-0.4.1/scripts/__init__.py +1 -0
  7. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/cli/clone_from_tsv.py +47 -13
  8. jmo_security-0.4.1/scripts/cli/jmo.py +1270 -0
  9. jmo_security-0.4.1/scripts/cli/jmotools.py +383 -0
  10. jmo_security-0.4.1/scripts/cli/wizard.py +857 -0
  11. jmo_security-0.4.1/scripts/core/__init__.py +1 -0
  12. jmo_security-0.4.1/scripts/core/adapters/__init__.py +1 -0
  13. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/adapters/bandit_adapter.py +14 -2
  14. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/adapters/checkov_adapter.py +5 -1
  15. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/adapters/gitleaks_adapter.py +18 -3
  16. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/adapters/hadolint_adapter.py +1 -0
  17. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/adapters/noseyparker_adapter.py +9 -3
  18. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/adapters/osv_adapter.py +15 -5
  19. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/adapters/semgrep_adapter.py +22 -5
  20. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/adapters/syft_adapter.py +9 -2
  21. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/adapters/tfsec_adapter.py +14 -4
  22. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/adapters/trivy_adapter.py +27 -5
  23. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/adapters/trufflehog_adapter.py +5 -1
  24. jmo_security-0.4.1/scripts/core/common_finding.py +126 -0
  25. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/config.py +29 -9
  26. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/generate_dashboard.py +248 -206
  27. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/normalize_and_report.py +29 -10
  28. jmo_security-0.4.1/scripts/core/reporters/__init__.py +1 -0
  29. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/reporters/basic_reporter.py +4 -1
  30. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/reporters/html_reporter.py +23 -9
  31. jmo_security-0.4.1/scripts/core/reporters/sarif_reporter.py +181 -0
  32. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/reporters/suppression_reporter.py +8 -2
  33. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/reporters/yaml_reporter.py +2 -2
  34. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/schema_utils.py +3 -1
  35. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/core/suppress.py +25 -8
  36. jmo_security-0.3.2/scripts/__init__.py +0 -1
  37. jmo_security-0.3.2/scripts/cli/jmo.py +0 -801
  38. jmo_security-0.3.2/scripts/cli/jmotools.py +0 -252
  39. jmo_security-0.3.2/scripts/core/__init__.py +0 -1
  40. jmo_security-0.3.2/scripts/core/adapters/__init__.py +0 -1
  41. jmo_security-0.3.2/scripts/core/common_finding.py +0 -41
  42. jmo_security-0.3.2/scripts/core/reporters/__init__.py +0 -1
  43. jmo_security-0.3.2/scripts/core/reporters/sarif_reporter.py +0 -64
  44. {jmo_security-0.3.2 → jmo_security-0.4.1}/LICENSE +0 -0
  45. {jmo_security-0.3.2 → jmo_security-0.4.1}/jmo_security.egg-info/dependency_links.txt +0 -0
  46. {jmo_security-0.3.2 → jmo_security-0.4.1}/jmo_security.egg-info/entry_points.txt +0 -0
  47. {jmo_security-0.3.2 → jmo_security-0.4.1}/jmo_security.egg-info/requires.txt +0 -0
  48. {jmo_security-0.3.2 → jmo_security-0.4.1}/jmo_security.egg-info/top_level.txt +0 -0
  49. {jmo_security-0.3.2 → jmo_security-0.4.1}/scripts/cli/__init__.py +0 -0
  50. {jmo_security-0.3.2 → jmo_security-0.4.1}/setup.cfg +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: jmo-security
3
- Version: 0.3.2
3
+ Version: 0.4.1
4
4
  Summary: JMo Security Audit Suite (terminal-first, multi-tool, unified outputs)
5
5
  Author-email: James Moceri <general@jmogaming.com>
6
6
  License: MIT
@@ -22,9 +22,12 @@ Dynamic: license-file
22
22
  </p>
23
23
 
24
24
  [![Tests](https://github.com/jimmy058910/jmo-security-repo/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/jimmy058910/jmo-security-repo/actions/workflows/tests.yml?query=branch%3Amain)
25
+ [![codecov](https://codecov.io/gh/jimmy058910/jmo-security-repo/branch/main/graph/badge.svg)](https://app.codecov.io/gh/jimmy058910/jmo-security-repo)
26
+ [![PyPI - Version](https://img.shields.io/pypi/v/jmo-security)](https://pypi.org/project/jmo-security/)
25
27
  [![Website](https://img.shields.io/website?url=https%3A%2F%2Fjmotools.com)](https://jmotools.com)
26
28
  [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
27
- [![Python](https://img.shields.io/badge/Python-%E2%89%A53.8-3776AB?logo=python&logoColor=white)](#)
29
+ [![Python](https://img.shields.io/badge/Python-%E2%89%A53.8-3776AB?logo=python&logoColor=white)](https://www.python.org/)
30
+
28
31
  [![Contributions welcome](https://img.shields.io/badge/Contributions-welcome-brightgreen.svg)](https://github.com/jimmy058910/jmo-security-repo/issues)
29
32
  [![Buy me an energy drink](https://img.shields.io/badge/Buy%20me%20an-energy%20drink-%23ff4d00)](https://ko-fi.com/jmogaming)
30
33
 
@@ -48,65 +51,173 @@ Project homepage: [jmotools.com](https://jmotools.com)
48
51
 
49
52
  > Origin story: This started as part of a Cybersecurity Capstone Project. It has since grown into a general-purpose toolkit. I’d love for folks with deeper expertise to jump in—issues and PRs are welcome!
50
53
 
51
- Thinking about contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, coding standards, and release steps.
54
+ Thinking about contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and coding standards. For publishing, see [docs/RELEASE.md](docs/RELEASE.md).
52
55
 
53
56
  Roadmap & history:
54
- - Completed steps (summary): see [RELEASE_NOTES.md](RELEASE_NOTES.md) → “Roadmap Summary (Steps 1–13)”
57
+
58
+ - **Latest:** ROADMAP #2 (Interactive Wizard) ✅ Complete - see [docs/examples/wizard-examples.md](docs/examples/wizard-examples.md)
59
+ - Completed steps (summary): see [CHANGELOG.md](CHANGELOG.md) → ROADMAP Items #1-2 and Steps 1–13
55
60
  - Active/planned work: see [ROADMAP.md](ROADMAP.md)
56
61
 
57
62
  For scanning a list of repos from a TSV end-to-end (clone + unshallow + full toolchain), see: [docs/examples/scan_from_tsv.md](docs/examples/scan_from_tsv.md)
58
63
 
59
- ## 🧪 New: Simple wrapper commands
64
+ ## CI and release at a glance
65
+
66
+ - Tests run on a matrix of operating systems and Python versions:
67
+ - OS: ubuntu-latest, macos-latest
68
+ - Python: 3.10, 3.11, 3.12
69
+ - CI uses concurrency to cancel redundant runs on rapid pushes and sets a 20-minute job timeout.
70
+ - Coverage is uploaded to Codecov without a token (OIDC/tokenless on public repos) using `codecov/codecov-action@v5`.
71
+ - Releases to PyPI use Trusted Publishers (OIDC) via `pypa/gh-action-pypi-publish@v1`; no PyPI API token is required once the repo is authorized in PyPI.
72
+
73
+ See `.github/workflows/tests.yml` and `.github/workflows/release.yml` for details.
74
+
75
+ Quick link: CI Troubleshooting → [Interpreting CI failures](docs/USER_GUIDE.md#interpreting-ci-failures-deeper-guide)
76
+
77
+ ## 🎉 Recent Improvements (Phase 1 - October 2025)
78
+
79
+ **Security & Bug Fixes:**
80
+
81
+ - ✅ **XSS vulnerability patched** in HTML dashboard with comprehensive input escaping
82
+ - ✅ **OSV scanner fully integrated** for open-source vulnerability detection
83
+ - ✅ **Type-safe severity enum** with comparison operators for cleaner code
84
+ - ✅ **Backward-compatible suppression keys** (`suppressions` and legacy `suppress`)
85
+
86
+ **Enhanced Features:**
87
+
88
+ - 🚀 **Enriched SARIF output** with CWE/OWASP/CVE taxonomies, code snippets, and CVSS scores
89
+ - ⚙️ **Configurable thread recommendations** via `jmo.yml` profiling section
90
+ - 📝 **Magic numbers extracted** to named constants for better maintainability
91
+ - 📚 **9 new roadmap enhancements** including Policy-as-Code (OPA), SLSA attestation, GitHub App, and more
92
+
93
+ **Quality Metrics:**
94
+
95
+ - ✅ 100/100 tests passing
96
+ - ✅ 88% code coverage (exceeds 85% requirement)
97
+ - ✅ No breaking changes to existing workflows
98
+
99
+ See [CHANGELOG.md](CHANGELOG.md) for complete details.
100
+
101
+ ## 🚀 Three Ways to Get Started
102
+
103
+ ### Option 1: 🧙 Interactive Wizard (Recommended for Beginners)
104
+
105
+ **Never used security scanners before?** Start with the guided wizard:
106
+
107
+ ```bash
108
+ jmotools wizard
109
+ ```
110
+
111
+ The wizard provides:
112
+
113
+ - ✅ **Step-by-step guidance** through all configuration options
114
+ - ✅ **Profile selection** (fast/balanced/deep) with time estimates
115
+ - ✅ **Docker vs native mode** - zero-installation Docker option!
116
+ - ✅ **Smart target detection** - auto-discovers repositories
117
+ - ✅ **Command preview** - see what will run before executing
118
+ - ✅ **Auto-open results** - dashboard and summary automatically displayed
119
+
120
+ **Non-interactive mode for automation:**
121
+
122
+ ```bash
123
+ jmotools wizard --yes # Use smart defaults
124
+ jmotools wizard --docker # Force Docker mode
125
+ ```
126
+
127
+ **Generate reusable artifacts:**
128
+
129
+ ```bash
130
+ jmotools wizard --emit-make-target Makefile.security # Team Makefile
131
+ jmotools wizard --emit-script scan.sh # Shell script
132
+ jmotools wizard --emit-gha .github/workflows/security.yml # GitHub Actions
133
+ ```
134
+
135
+ 📖 **Full wizard guide:** [docs/examples/wizard-examples.md](docs/examples/wizard-examples.md)
60
136
 
61
- If you're just getting started, use the beginner-friendly wrapper:
137
+ ---
62
138
 
63
- - `jmotools fast` quick scan (fast profile)
64
- - `jmotools balanced` — default balanced scan
65
- - `jmotools full` — deep scan with more tools
139
+ ### Option 2: 🐳 Docker (Zero Installation)
66
140
 
67
- Examples:
141
+ **Want to start scanning immediately with no tool installation?**
68
142
 
69
143
  ```bash
70
- # Fast profile over repos listed in a TSV (auto-clone) and open dashboard when done
71
- jmotools fast --tsv ai-search/candidates.tsv --dest ./repos-tsv
144
+ # Pull the image (one-time, ~500MB)
145
+ docker pull ghcr.io/jimmy058910/jmo-security:latest
146
+
147
+ # Scan current directory
148
+ docker run --rm -v $(pwd):/scan ghcr.io/jimmy058910/jmo-security:latest \
149
+ scan --repo /scan --results /scan/results --profile balanced --human-logs
72
150
 
73
- # Full (deep) profile targeting an existing directory of repos
74
- jmotools full --repos-dir ./repos-tsv --allow-missing-tools
151
+ # View results
152
+ open results/summaries/dashboard.html # macOS
153
+ xdg-open results/summaries/dashboard.html # Linux
75
154
  ```
76
155
 
77
- Under the hood, these commands verify your OS/tools, optionally clone from a TSV, run `jmo ci` with the appropriate profile, and open `dashboard.html` and `SUMMARY.md` on completion.
156
+ **Three image variants available:**
157
+
158
+ - `latest` (~500MB) - All 11+ scanners included
159
+ - `slim` (~200MB) - Core 6 scanners for CI/CD
160
+ - `alpine` (~150MB) - Minimal footprint
161
+
162
+ 📖 **Complete Docker guide:** [docs/DOCKER_README.md](docs/DOCKER_README.md)
163
+ 📖 **Beginner Docker tutorial:** [docs/DOCKER_QUICKSTART_BEGINNERS.md](docs/DOCKER_QUICKSTART_BEGINNERS.md)
164
+
165
+ ---
166
+
167
+ ### Option 3: 🧪 CLI Wrapper Commands (Local Install)
168
+
169
+ **Already have tools installed? Use our simple wrapper commands:**
170
+
171
+ ```bash
172
+ # Quick fast scan (auto-opens results)
173
+ jmotools fast --repos-dir ~/repos
174
+
175
+ # Balanced scan (recommended default)
176
+ jmotools balanced --repos-dir ~/repos
177
+
178
+ # Deep scan with all tools
179
+ jmotools full --repos-dir ~/repos
180
+ ```
78
181
 
79
- Setup tools quickly:
182
+ **Clone from TSV and scan:**
80
183
 
81
184
  ```bash
82
- # Check tools or auto-install on Linux/WSL (where supported)
83
- jmotools setup --check
84
- jmotools setup --auto-install # or --print-commands, --force-reinstall
185
+ jmotools balanced --tsv ./repositories.tsv --dest ./cloned-repos
85
186
  ```
86
187
 
87
- Makefile shortcuts:
188
+ **Setup tools quickly:**
88
189
 
89
190
  ```bash
90
- make setup # jmotools setup --check (installs package if needed)
91
- make fast DIR=~/repos # jmotools fast --repos-dir ~/repos
191
+ jmotools setup --check # Verify tool installation
192
+ jmotools setup --auto-install # Auto-install on Linux/WSL/macOS
193
+ ```
194
+
195
+ **Makefile shortcuts:**
196
+
197
+ ```bash
198
+ make setup # Verify tools (installs package if needed)
199
+ make fast DIR=~/repos # Run fast profile
92
200
  make balanced DIR=~/repos
93
201
  make full DIR=~/repos
94
202
  ```
95
203
 
204
+ Note: Under the hood, wrapper commands verify your OS/tools, optionally clone from TSV, run `jmo ci` with the appropriate profile, and auto-open results.
205
+
96
206
  ## 🎯 Overview
97
207
 
98
208
  This project provides an automated framework for conducting thorough security audits on code repositories. It orchestrates multiple industry-standard security tools to detect secrets, vulnerabilities, and security issues.
99
209
 
100
210
  ### Key Features
101
211
 
102
- - ✅ **Multi-Tool Scanning**: Curated set covering secrets (gitleaks, noseyparker), SAST (semgrep, bandit), SBOM+vuln/misconfig (syft+trivy), IaC (checkov, tfsec), Dockerfile (hadolint)
103
- - 📊 **Comprehensive Reporting**: Unified findings (JSON/YAML), SARIF, Markdown summary, and an interactive HTML dashboard
104
- - 🎨 **Easy-to-Read Outputs**: Well-formatted reports with severity categorization
212
+ - ✅ **Multi-Tool Scanning**: Curated set covering secrets (gitleaks, noseyparker, trufflehog), SAST (semgrep, bandit), SBOM+vuln/misconfig (syft+trivy), IaC (checkov, tfsec), Dockerfile (hadolint), and open-source vulnerabilities (osv-scanner)
213
+ - 📊 **Comprehensive Reporting**: Unified findings (JSON/YAML), enriched SARIF 2.1.0 with taxonomies, Markdown summary, and an interactive HTML dashboard with XSS protection
214
+ - 🎨 **Easy-to-Read Outputs**: Well-formatted reports with severity categorization using type-safe enums
105
215
  - 🔄 **Automated Workflows**: One CLI to scan, aggregate, and gate on severity (scan/report/ci)
106
- - 🧭 **Profiles and Overrides**: Named profiles, per-tool flags/timeouts, include/exclude patterns
216
+ - 🧭 **Profiles and Overrides**: Named profiles, per-tool flags/timeouts, include/exclude patterns, configurable thread recommendations
107
217
  - 🔁 **Resilience**: Timeouts, retries with per-tool success codes, human-friendly logs, graceful cancel
218
+ - 🔒 **Security-First**: XSS vulnerability patched, comprehensive input escaping, secure-by-default configurations
108
219
 
109
- ## 🚀 Quick Start
220
+ ## 🚀 Quick Start (Local Installation)
110
221
 
111
222
  ### Install or Update (curated tools)
112
223
 
@@ -125,21 +236,46 @@ Optional: install the package locally to get `jmo` and `jmotools` commands on yo
125
236
  pip install -e .
126
237
  ```
127
238
 
239
+ #### Pre-commit hooks (YAML & Actions validation)
240
+
241
+ We ship pre-commit hooks for YAML linting and GitHub Actions validation (among other basic checks):
242
+
243
+ ```bash
244
+ make pre-commit-install # installs the git hooks
245
+ make pre-commit-run # run checks on all files
246
+ ```
247
+
248
+ These run locally via pre-commit and are also enforced in CI.
249
+
250
+ We ship a `.yamllint.yaml` and validate GitHub Actions workflows via `actionlint`. The same checks are executed in CI.
251
+
252
+ #### Reproducible dev dependencies (optional)
253
+
254
+ This repo ships a `requirements-dev.in` with a compiled `requirements-dev.txt`. Use pip-tools or uv to pin/sync your dev environment:
255
+
256
+ ```bash
257
+ make upgrade-pip
258
+ make deps-compile # compile dev deps
259
+ make deps-sync # sync env to compiled lock
260
+ ```
261
+
262
+ CI verifies that `requirements-dev.txt` is up to date on PRs. If it fails, run `make deps-compile` and commit the diff.
263
+
128
264
  ### Quick Start (Unified CLI)
129
265
 
130
- 1) Verify your environment (Linux/WSL/macOS) and see install hints for optional tools:
266
+ 1. Verify your environment (Linux/WSL/macOS) and see install hints for optional tools:
131
267
 
132
268
  ```bash
133
269
  make verify-env
134
270
  ```
135
271
 
136
- 2) Install Python dev dependencies (for running tests and reporters):
272
+ 1. Install Python dev dependencies (for running tests and reporters):
137
273
 
138
274
  ```bash
139
275
  make dev-deps
140
276
  ```
141
277
 
142
- 3) Scan repositories using a profile, then aggregate reports:
278
+ 1. Scan repositories using a profile, then aggregate reports:
143
279
 
144
280
  ```bash
145
281
  # Scan immediate subfolders under ~/repos with the 'balanced' profile (default)
@@ -150,8 +286,11 @@ python3 scripts/cli/jmo.py scan --repos-dir ~/repos --profile-name balanced --hu
150
286
  python3 scripts/cli/jmo.py report ./results --profile --human-logs
151
287
  # or
152
288
  python3 scripts/cli/jmo.py report --results-dir ./results --profile --human-logs
289
+ ```
290
+
291
+ #### Or do both in one step for CI with a failure threshold
153
292
 
154
- # Or do both in one step for CI with a failure threshold
293
+ ```bash
155
294
  python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name fast --fail-on HIGH --profile --human-logs
156
295
  ```
157
296
 
@@ -196,9 +335,9 @@ Prefer the Python CLI above. For legacy flows, you can still use the shell wrapp
196
335
  python3 scripts/cli/jmo.py ci --repos-dir ~/test-repos --fail-on HIGH --profile --human-logs
197
336
 
198
337
  # 3. View results
199
- cat ~/security-results-*/SUMMARY_REPORT.md
200
- # macOS: open ~/security-results-*/dashboard.html
201
- # Linux: xdg-open ~/security-results-*/dashboard.html
338
+ cat results/summaries/SUMMARY.md
339
+ # macOS: open results/summaries/dashboard.html
340
+ # Linux: xdg-open results/summaries/dashboard.html
202
341
  ```
203
342
 
204
343
  Looking for screenshots and how to capture them? See: [docs/screenshots/README.md](docs/screenshots/README.md)
@@ -215,53 +354,44 @@ The security audit follows this workflow:
215
354
  4. **Report Generation**: JSON/MD/YAML/HTML/SARIF and suppression summary
216
355
  5. **Dashboard Creation**: Self-contained HTML dashboard with an optional profiling panel
217
356
 
218
- ### Output Structure
219
-
220
- ```
221
- security-results-YYYYMMDD-HHMMSS/
222
- ├── SUMMARY_REPORT.md # Executive summary
223
- ├── dashboard.html # Interactive HTML dashboard
224
- ├── individual-repos/ # Per-repository results
225
- └── [repo-name]/
226
- │ ├── README.md # Formatted findings report
227
- │ ├── gitleaks.json # Gitleaks raw output
228
- │ ├── trufflehog.json # TruffleHog raw output
229
- │ ├── semgrep.json # Semgrep raw output
230
- │ ├── noseyparker.json # Nosey Parker raw output
231
- └── *.log # Tool execution logs
232
- ├── raw-outputs/ # Compressed per-repo raw artifacts
233
- ├── tool-comparisons/
234
- └── comparison.md # Tool performance comparison
235
- ├── summaries/
236
- └── metrics.csv # Aggregated metrics
237
-
238
- ```
239
-
240
- ### Report Types
241
-
242
- 1. **Summary Report** (`SUMMARY_REPORT.md`)
243
- - Executive summary
244
- - Aggregate statistics
245
- - Repository breakdown table
246
- - Prioritized recommendations
247
-
248
- 2. **HTML Dashboard** (`dashboard.html`)
249
- - Visual metrics cards
250
- - Severity breakdown
251
- - Repository comparison table
252
- - Tool performance analysis
253
-
254
- 3. **Individual Reports** (`individual-repos/*/README.md`)
255
- - Repository-specific findings
256
- - Tool-by-tool breakdown
257
- - Detailed issue listings
258
- - Severity classifications
259
-
260
- 4. **Tool Comparison** (`tool-comparisons/comparison.md`)
261
- - Detection metrics
262
- - Tool capabilities matrix
263
- - Implementation strategy guide
264
- - Tool selection recommendations
357
+ ### Output Structure (Default)
358
+
359
+ ```text
360
+ results/
361
+ ├── individual-repos/
362
+ │ └── <repo-name>/
363
+ ├── gitleaks.json
364
+ ├── trufflehog.json
365
+ │ ├── semgrep.json
366
+ │ ├── noseyparker.json
367
+ │ ├── syft.json
368
+ │ ├── trivy.json
369
+ │ ├── hadolint.json
370
+ ├── checkov.json
371
+ ├── tfsec.json
372
+ │ └── bandit.json
373
+ └── summaries/
374
+ ├── findings.json
375
+ ├── findings.yaml # requires PyYAML
376
+ ├── findings.sarif # SARIF 2.1.0
377
+ ├── SUMMARY.md
378
+ ├── dashboard.html
379
+ ├── SUPPRESSIONS.md # written when suppressions apply
380
+ └── timings.json # written when --profile is used
381
+ ```
382
+
383
+ ### Reporters
384
+
385
+ The aggregator writes unified outputs under `results/summaries/`:
386
+
387
+ - JSON (`findings.json`) — complete, machine-readable findings list
388
+ - Markdown (`SUMMARY.md`) — human-readable overview with severity counts and top rules
389
+ - YAML (`findings.yaml`) — optional; requires PyYAML
390
+ - HTML (`dashboard.html`) — interactive dashboard with filters, sorting, exports, and theme toggle
391
+ - SARIF (`findings.sarif`) — 2.1.0 for code scanning integrations
392
+ - Suppression summary (`SUPPRESSIONS.md`) — appears when suppression rules filter findings
393
+
394
+ See `SAMPLE_OUTPUTS.md` for real examples produced from the `infra-demo` fixture.
265
395
 
266
396
  ### How we normalize findings
267
397
 
@@ -271,8 +401,6 @@ All tool outputs are converted into a single CommonFinding schema during aggrega
271
401
  - Required fields include: schemaVersion (1.0.0), id, ruleId, severity, tool (name/version), location (path/lines), and message. Optional fields include title, description, remediation, references, tags, cvss, and raw (original tool payload).
272
402
  - Fingerprint (id): deterministically derived from a stable subset of attributes (tool | ruleId | path | startLine | message snippet) to support cross-tool dedupe. The aggregation step deduplicates by this id.
273
403
 
274
-
275
-
276
404
  ## 🛠️ Tool Installation
277
405
 
278
406
  ### macOS (Homebrew)
@@ -319,10 +447,10 @@ curl -sSfL https://raw.githubusercontent.com/trufflesecurity/trufflehog/main/scr
319
447
 
320
448
  Nosey Parker doesn’t ship via apt/brew universally. Install the release binary and put it on your PATH:
321
449
 
322
- 1) Download the latest release for your OS/arch from:
323
- https://github.com/praetorian-inc/noseyparker/releases
450
+ 1. Download the latest release for your OS/arch from:
451
+ <https://github.com/praetorian-inc/noseyparker/releases>
324
452
 
325
- 2) Unpack and move the binary onto PATH (example for Linux x86_64):
453
+ 2. Unpack and move the binary onto PATH (example for Linux x86_64):
326
454
 
327
455
  ```bash
328
456
  tar -xzf noseyparker-*.tar.gz
@@ -339,11 +467,12 @@ On WSL Ubuntu, installing Nosey Parker natively is the most reliable path (prebu
339
467
 
340
468
  The CLI automatically falls back to a Docker-based Nosey Parker runner when the local binary is missing or not runnable (common on older WSL/glibc). When enabled via profiles, scans will transparently produce the expected JSON here:
341
469
 
342
- ```
470
+ ```text
343
471
  results/individual-repos/<repo-name>/noseyparker.json
344
472
  ```
345
473
 
346
474
  Requirements for the fallback:
475
+
347
476
  - Docker installed and running
348
477
  - Ability to pull or use `ghcr.io/praetorian-inc/noseyparker:latest`
349
478
 
@@ -380,6 +509,7 @@ Note: we recommend isolating CLI tools via pipx or OS packages for stability. Th
380
509
  This helper script streamlines the process of cloning multiple repositories for security scanning, with performance optimizations for WSL environments.
381
510
 
382
511
  **Features:**
512
+
383
513
  - 🚀 Shallow clones (depth=1) for faster cloning
384
514
  - ⚡ Parallel cloning for improved performance
385
515
  - 🔄 Unshallow option for secret scanners requiring full history
@@ -405,7 +535,8 @@ This helper script streamlines the process of cloning multiple repositories for
405
535
  ```
406
536
 
407
537
  **Repository List Format (`samples/repos.txt`):**
408
- ```
538
+
539
+ ```text
409
540
  # One GitHub repository URL per line
410
541
  # Lines starting with # are comments
411
542
  https://github.com/user/repo1.git
@@ -413,37 +544,29 @@ https://github.com/user/repo2.git
413
544
  ```
414
545
 
415
546
  **Performance Tips for WSL:**
547
+
416
548
  1. Use shallow clones initially for 10x faster cloning
417
549
  2. Adjust `--parallel` based on network speed (default: 4)
418
550
  3. Use `--unshallow` only if secret scanners need full git history
419
551
  4. Clone to WSL filesystem (not Windows mount) for better performance
420
552
 
421
- ### Running Individual Scripts
553
+ ### CLI-first usage
422
554
 
423
- 1. **Tool Check Only**:
424
- ```bash
425
- ./scripts/core/check_tools.sh
426
- ```
555
+ Prefer the Python CLI for report generation from existing results:
427
556
 
428
- 2. **Main Audit Script**:
429
557
  ```bash
430
- ./scripts/core/run_security_audit.sh [testing_directory] [output_directory]
431
- ```
558
+ # Default reporters (formats controlled by jmo.yml)
559
+ python3 scripts/cli/jmo.py report /path/to/results
432
560
 
433
- 3. **Generate Dashboard Only**:
434
- ```bash
435
- # Generate dashboard with default output (results_dir/dashboard.html)
436
- python3 scripts/core/generate_dashboard.py /path/to/results
561
+ # Set thread workers explicitly for aggregation
562
+ python3 scripts/cli/jmo.py report /path/to/results --threads 6
437
563
 
438
- # Generate dashboard with custom output path
439
- python3 scripts/core/generate_dashboard.py /path/to/results /custom/path/dashboard.html
440
- ```
564
+ # Record profiling timings (writes summaries/timings.json)
565
+ python3 scripts/cli/jmo.py report /path/to/results --profile
441
566
 
442
- The dashboard generator supports:
443
- - Multiple TruffleHog output formats (JSON arrays, NDJSON, single objects, empty files)
444
- - Automatic parent directory creation for custom output paths
445
- - Graceful handling of missing or empty scan results
446
- - UTF-8 safe file I/O for international characters
567
+ # Human-friendly colored logs (stderr)
568
+ python3 scripts/cli/jmo.py report /path/to/results --human-logs
569
+ ```
447
570
 
448
571
  ### Unified CLI: report-only
449
572
 
@@ -486,7 +609,7 @@ python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name balanced --fail
486
609
 
487
610
  The `summaries/` folder also contains unified outputs:
488
611
 
489
- ```
612
+ ```text
490
613
  summaries/
491
614
  ├── findings.json # Unified normalized findings (machine-readable)
492
615
  ├── SUMMARY.md # Human-readable summary
@@ -503,7 +626,7 @@ You can define named profiles in `jmo.yml` to control which tools run, include/e
503
626
 
504
627
  Example `jmo.yml` snippet:
505
628
 
506
- ```
629
+ ```yaml
507
630
  default_profile: fast
508
631
  retries: 1
509
632
  profiles:
@@ -530,7 +653,7 @@ per_tool:
530
653
 
531
654
  Using a profile from CLI:
532
655
 
533
- ```
656
+ ```bash
534
657
  # Scan using profile 'fast' with human-friendly logs
535
658
  python3 scripts/cli/jmo.py scan --repos-dir ~/repos --profile-name fast --human-logs
536
659
 
@@ -539,16 +662,12 @@ python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name deep --fail-on
539
662
  ```
540
663
 
541
664
  Retries behavior:
665
+
542
666
  - Global `retries` (or per-profile) retries failed tool commands a limited number of times
543
667
  - Some tools use non-zero exit to indicate “findings”; we treat those as success codes to avoid useless retries
544
668
 
545
669
  Human logs show per-tool retry attempts when > 1, e.g.: `attempts={'semgrep': 2}`
546
670
 
547
- 4. **Generate Comparison Report**:
548
- ```bash
549
- ./scripts/core/generate_comparison_report.sh /path/to/results
550
- ```
551
-
552
671
  ### Customizing Tool Execution
553
672
 
554
673
  Prefer jmo.yml profiles and per_tool overrides. For one-off local tweaks, use:
@@ -567,10 +686,13 @@ python3 scripts/cli/jmo.py scan --repos-dir ~/repos --tools gitleaks semgrep --t
567
686
 
568
687
  ### Severity Levels
569
688
 
689
+ The toolkit uses a type-safe severity enum with comparison operators for consistent filtering and sorting:
690
+
570
691
  - **CRITICAL**: Verified secrets requiring immediate action
571
692
  - **HIGH**: Likely secrets or serious vulnerabilities
572
693
  - **MEDIUM**: Potential issues requiring review
573
- - **LOW/INFO**: Informational findings
694
+ - **LOW**: Minor issues for regular maintenance
695
+ - **INFO**: Informational findings
574
696
 
575
697
  ### Key Metrics
576
698
 
@@ -589,46 +711,26 @@ python3 scripts/cli/jmo.py scan --repos-dir ~/repos --tools gitleaks semgrep --t
589
711
  ## 🎯 Three-Stage Implementation Strategy
590
712
 
591
713
  ### Stage 1: Pre-commit Hooks
714
+
592
715
  - **Tool**: Gitleaks
593
716
  - **Purpose**: Prevent secrets before commit
594
717
  - **Speed**: Fast (suitable for developer workflow)
595
718
 
596
719
  ### Stage 2: CI/CD Pipeline
720
+
597
721
  - **Tools**: Gitleaks + Semgrep
598
722
  - **Purpose**: Automated PR/commit scanning
599
723
  - **Coverage**: Secrets + vulnerabilities
600
724
 
601
725
  ### Stage 3: Deep Periodic Audits
726
+
602
727
  - **Tools**: All tools
603
728
  - **Purpose**: Comprehensive security assessment
604
729
  - **Frequency**: Weekly/monthly
605
730
 
606
- ## 📊 Sample Output
607
-
608
- ### Dashboard Preview
609
- The HTML dashboard provides:
610
- - Visual metric cards with key statistics
611
- - Severity breakdown tables
612
- - Repository-by-repository comparison
613
- - Tool performance analysis
614
- - Actionable recommendations
615
-
616
- ### Sample Summary Report
617
- ```markdown
618
- ## Aggregate Results
731
+ ## 📊 Sample Outputs
619
732
 
620
- ### Overall Statistics
621
- - Total Issues Found: 1562
622
- - Critical Issues: 5
623
- - High Severity Issues: 579
624
- - Medium Severity Issues: 61
625
- - Verified Secrets: 5
626
-
627
- ### Recommendations
628
- - Rotate all 5 verified secrets immediately
629
- - Prioritize remediation of 584 critical and high severity issues
630
- - Schedule follow-up review for the remaining medium findings
631
- ```
733
+ For a current snapshot produced from the `infra-demo` fixture, see: [SAMPLE_OUTPUTS.md](SAMPLE_OUTPUTS.md).
632
734
 
633
735
  ## 🤝 Contributing
634
736
 
@@ -638,7 +740,7 @@ Contributions are welcome! Please feel free to submit pull requests or open issu
638
740
 
639
741
  If this toolkit saves you time, consider fueling development with an energy drink.
640
742
 
641
- - Prefer one-time tips? Ko‑fi: https://ko-fi.com/jmogaming
743
+ - Prefer one-time tips? Ko‑fi: <https://ko-fi.com/jmogaming>
642
744
  - When you’re ready, replace the badge target with your preferred platform: GitHub Sponsors (industry standard), Open Collective, Ko-fi, or Stripe Checkout.
643
745
  - GitHub Sponsors integrates directly with your GitHub profile and repository sidebar once enabled.
644
746
 
@@ -651,9 +753,6 @@ MIT License. See LICENSE.
651
753
  - [Gitleaks Documentation](https://github.com/zricethezav/gitleaks)
652
754
  - [TruffleHog Documentation](https://github.com/trufflesecurity/trufflehog)
653
755
  - [Semgrep Documentation](https://semgrep.dev)
654
- - [Nosey Parker Documentation](https://github.com/praetorian-inc/noseyparker)
655
-
656
- ## 💡 Tips
657
756
 
658
757
  1. **Start Small**: Test on a single repository first
659
758
  2. **Review Regularly**: Schedule periodic audits
@@ -666,22 +765,36 @@ MIT License. See LICENSE.
666
765
  ### Common Issues
667
766
 
668
767
  **Problem**: Tools not found
669
- - **Solution**: Run `./scripts/cli/security_audit.sh --check` to verify installation
768
+
769
+ - **Solution**: Run `make verify-env` (or `jmotools setup --check`) to verify installation and get platform-specific hints
670
770
 
671
771
  **Problem**: JSON parsing errors
772
+
672
773
  - **Solution**: Ensure jq is installed and tools are outputting valid JSON
673
774
 
674
775
  **Problem**: Permission denied
675
- - **Solution**: Run `chmod +x *.sh` to make scripts executable
776
+
777
+ - **Solution**: Ensure scripts are executable:
778
+
779
+ ```bash
780
+ find scripts -type f -name "*.sh" -exec chmod +x {} +
781
+ ```
676
782
 
677
783
  **Problem**: Out of memory
784
+
678
785
  - **Solution**: Scan repositories in smaller batches
679
786
 
787
+ ```bash
788
+ ./scripts/core/populate_targets.sh --unshallow
789
+ ```
790
+
680
791
  **Problem**: Path errors (e.g., "//run_security_audit.sh not found")
792
+
681
793
  - **Solution**: This issue has been fixed in the latest version. Update to the latest main branch.
682
794
  - The wrapper scripts now use absolute paths computed from the script's real path location.
683
795
 
684
796
  **Problem**: AttributeError when generating dashboard with TruffleHog results
797
+
685
798
  - **Solution**: This has been fixed. The dashboard generator now handles all TruffleHog output formats:
686
799
  - JSON arrays: `[{...}, {...}]`
687
800
  - Single objects: `{...}`
@@ -705,11 +818,12 @@ python3 scripts/core/generate_dashboard.py ~/security-results-20251010-120000 ~/
705
818
  ```
706
819
 
707
820
  This is useful when you want to:
821
+
708
822
  - Update the dashboard after manually editing JSON files
709
823
  - Generate multiple dashboards with different configurations
710
824
  - Share results by exporting to a specific location
711
825
 
712
826
  ---
713
827
 
714
- **Last Updated**: October 10th, 2025
828
+ **Last Updated**: October 13th, 2025
715
829
  **Author**: James Moceri