jmo-security 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of jmo-security might be problematic. Click here for more details.
- jmo_security-0.3.0/LICENSE +21 -0
- jmo_security-0.3.0/PKG-INFO +632 -0
- jmo_security-0.3.0/README.md +616 -0
- jmo_security-0.3.0/jmo_security.egg-info/PKG-INFO +632 -0
- jmo_security-0.3.0/jmo_security.egg-info/SOURCES.txt +10 -0
- jmo_security-0.3.0/jmo_security.egg-info/dependency_links.txt +1 -0
- jmo_security-0.3.0/jmo_security.egg-info/entry_points.txt +2 -0
- jmo_security-0.3.0/jmo_security.egg-info/requires.txt +4 -0
- jmo_security-0.3.0/jmo_security.egg-info/top_level.txt +1 -0
- jmo_security-0.3.0/pyproject.toml +28 -0
- jmo_security-0.3.0/scripts/__init__.py +1 -0
- jmo_security-0.3.0/setup.cfg +4 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 jimmy058910
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,632 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: jmo-security
|
|
3
|
+
Version: 0.3.0
|
|
4
|
+
Summary: JMo Security Audit Suite (terminal-first, multi-tool, unified outputs)
|
|
5
|
+
Author-email: James Moceri <general@jmogaming.com>
|
|
6
|
+
License: MIT
|
|
7
|
+
Classifier: Programming Language :: Python :: 3
|
|
8
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
9
|
+
Classifier: Operating System :: OS Independent
|
|
10
|
+
Requires-Python: >=3.8
|
|
11
|
+
Description-Content-Type: text/markdown
|
|
12
|
+
License-File: LICENSE
|
|
13
|
+
Provides-Extra: reporting
|
|
14
|
+
Requires-Dist: PyYAML>=6.0; extra == "reporting"
|
|
15
|
+
Requires-Dist: jsonschema>=4.0; extra == "reporting"
|
|
16
|
+
|
|
17
|
+
# JMo's Security Audit Tool Suite
|
|
18
|
+
|
|
19
|
+
<p align="center">
|
|
20
|
+
<img src="assets/jmo-logo.png" alt="JMo Security Audit Tool Suite" width="220" />
|
|
21
|
+
</p>
|
|
22
|
+
|
|
23
|
+
[](https://github.com/jimmy058910/jmo-security-repo/actions/workflows/tests.yml)
|
|
24
|
+
[](https://codecov.io/gh/jimmy058910/jmo-security-repo)
|
|
25
|
+
[](https://pypi.org/project/jmo-security/)
|
|
26
|
+
|
|
27
|
+
A terminal-first, cross-platform security audit toolkit that orchestrates multiple scanners (secrets, SAST, SBOM, IaC, Dockerfile) with a unified Python CLI, normalized outputs, and an HTML dashboard.
|
|
28
|
+
|
|
29
|
+
👉 New here? Read the comprehensive User Guide: [docs/USER_GUIDE.md](docs/USER_GUIDE.md)
|
|
30
|
+
Docs hub: [docs/index.md](docs/index.md)
|
|
31
|
+
|
|
32
|
+
## 🎯 Overview
|
|
33
|
+
|
|
34
|
+
This project provides an automated framework for conducting thorough security audits on code repositories. It orchestrates multiple industry-standard security tools to detect secrets, vulnerabilities, and security issues.
|
|
35
|
+
|
|
36
|
+
### Key Features
|
|
37
|
+
|
|
38
|
+
- ✅ **Multi-Tool Scanning**: Curated set covering secrets (gitleaks, noseyparker), SAST (semgrep, bandit), SBOM+vuln/misconfig (syft+trivy), IaC (checkov, tfsec), Dockerfile (hadolint)
|
|
39
|
+
- 📊 **Comprehensive Reporting**: Unified findings (JSON/YAML), SARIF, Markdown summary, and an interactive HTML dashboard
|
|
40
|
+
- 🎨 **Easy-to-Read Outputs**: Well-formatted reports with severity categorization
|
|
41
|
+
- 🔄 **Automated Workflows**: One CLI to scan, aggregate, and gate on severity (scan/report/ci)
|
|
42
|
+
- 🧭 **Profiles and Overrides**: Named profiles, per-tool flags/timeouts, include/exclude patterns
|
|
43
|
+
- 🔁 **Resilience**: Timeouts, retries with per-tool success codes, human-friendly logs, graceful cancel
|
|
44
|
+
|
|
45
|
+
## 🚀 Quick Start
|
|
46
|
+
|
|
47
|
+
### Install or Update (curated tools)
|
|
48
|
+
|
|
49
|
+
These targets detect Linux/WSL/macOS and install or upgrade the curated CLI tools used by this suite. They also surface helpful hints if a platform step needs manual action.
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
make tools # one-time install of curated tools
|
|
53
|
+
make tools-upgrade # refresh/upgrade curated tools
|
|
54
|
+
make verify-env # check OS/WSL/macOS and tool availability
|
|
55
|
+
make dev-deps # install Python dev dependencies
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
### Quick Start (Unified CLI)
|
|
59
|
+
|
|
60
|
+
1) Verify your environment (Linux/WSL/macOS) and see install hints for optional tools:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
make verify-env
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
2) Install Python dev dependencies (for running tests and reporters):
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
make dev-deps
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
3) Scan repositories using a profile, then aggregate reports:
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
# Scan immediate subfolders under ~/repos with the 'balanced' profile (default)
|
|
76
|
+
python3 scripts/cli/jmo.py scan --repos-dir ~/repos --profile-name balanced --human-logs
|
|
77
|
+
|
|
78
|
+
# Aggregate and write unified outputs to results/summaries
|
|
79
|
+
python3 scripts/cli/jmo.py report ./results --profile --human-logs
|
|
80
|
+
|
|
81
|
+
# Or do both in one step for CI with a failure threshold
|
|
82
|
+
python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name fast --fail-on HIGH --profile --human-logs
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Outputs include: summaries/findings.json, SUMMARY.md, findings.yaml, findings.sarif (enabled by default), dashboard.html, and timings.json (when profiling).
|
|
86
|
+
|
|
87
|
+
### Basic Usage
|
|
88
|
+
|
|
89
|
+
#### Optional: Quick Setup with Helper Script
|
|
90
|
+
|
|
91
|
+
Use the `populate_targets.sh` helper script to clone multiple repositories for testing (optimized for WSL):
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
# Clone sample vulnerable repos (fast shallow clones)
|
|
95
|
+
./scripts/core/populate_targets.sh
|
|
96
|
+
|
|
97
|
+
# Clone from custom list with full history
|
|
98
|
+
./scripts/core/populate_targets.sh --list my-repos.txt --full
|
|
99
|
+
|
|
100
|
+
# Clone with 8 parallel jobs for faster performance
|
|
101
|
+
./scripts/core/populate_targets.sh --parallel 8
|
|
102
|
+
|
|
103
|
+
# Unshallow repos if secret scanners need full git history
|
|
104
|
+
./scripts/core/populate_targets.sh --unshallow
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
#### Running Security Scans (legacy shell script)
|
|
108
|
+
|
|
109
|
+
Prefer the Python CLI above. For legacy flows, you can still use the shell wrapper:
|
|
110
|
+
|
|
111
|
+
```bash
|
|
112
|
+
./scripts/cli/security_audit.sh -d ~/security-testing # scan
|
|
113
|
+
./scripts/cli/security_audit.sh --check # verify tools
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
#### End-to-End Workflow
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
# 1. Clone test repositories (shallow for speed)
|
|
120
|
+
./scripts/core/populate_targets.sh --dest ~/test-repos --parallel 4
|
|
121
|
+
|
|
122
|
+
# 2. Run security audit (preferred)
|
|
123
|
+
python3 scripts/cli/jmo.py ci --repos-dir ~/test-repos --fail-on HIGH --profile --human-logs
|
|
124
|
+
|
|
125
|
+
# 3. View results
|
|
126
|
+
cat ~/security-results-*/SUMMARY_REPORT.md
|
|
127
|
+
# macOS: open ~/security-results-*/dashboard.html
|
|
128
|
+
# Linux: xdg-open ~/security-results-*/dashboard.html
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
## 📚 Documentation
|
|
132
|
+
|
|
133
|
+
### Workflow (at a glance)
|
|
134
|
+
|
|
135
|
+
The security audit follows this workflow:
|
|
136
|
+
|
|
137
|
+
1. **Tool Verification**: Checks all required tools are installed
|
|
138
|
+
2. **Repository Scanning**: jmo scan orchestrates tools per jmo.yml (profiles, overrides, retries)
|
|
139
|
+
3. **Results Aggregation**: jmo report normalizes tool outputs to a CommonFinding shape
|
|
140
|
+
4. **Report Generation**: JSON/MD/YAML/HTML/SARIF and suppression summary
|
|
141
|
+
5. **Dashboard Creation**: Self-contained HTML dashboard with an optional profiling panel
|
|
142
|
+
|
|
143
|
+
### Output Structure
|
|
144
|
+
|
|
145
|
+
```
|
|
146
|
+
security-results-YYYYMMDD-HHMMSS/
|
|
147
|
+
├── SUMMARY_REPORT.md # Executive summary
|
|
148
|
+
├── dashboard.html # Interactive HTML dashboard
|
|
149
|
+
├── individual-repos/ # Per-repository results
|
|
150
|
+
│ └── [repo-name]/
|
|
151
|
+
│ ├── README.md # Formatted findings report
|
|
152
|
+
│ ├── gitleaks.json # Gitleaks raw output
|
|
153
|
+
│ ├── trufflehog.json # TruffleHog raw output
|
|
154
|
+
│ ├── semgrep.json # Semgrep raw output
|
|
155
|
+
│ ├── noseyparker.json # Nosey Parker raw output
|
|
156
|
+
│ └── *.log # Tool execution logs
|
|
157
|
+
├── raw-outputs/ # Compressed per-repo raw artifacts
|
|
158
|
+
├── tool-comparisons/
|
|
159
|
+
│ └── comparison.md # Tool performance comparison
|
|
160
|
+
├── summaries/
|
|
161
|
+
│ └── metrics.csv # Aggregated metrics
|
|
162
|
+
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
### Report Types
|
|
166
|
+
|
|
167
|
+
1. **Summary Report** (`SUMMARY_REPORT.md`)
|
|
168
|
+
- Executive summary
|
|
169
|
+
- Aggregate statistics
|
|
170
|
+
- Repository breakdown table
|
|
171
|
+
- Prioritized recommendations
|
|
172
|
+
|
|
173
|
+
2. **HTML Dashboard** (`dashboard.html`)
|
|
174
|
+
- Visual metrics cards
|
|
175
|
+
- Severity breakdown
|
|
176
|
+
- Repository comparison table
|
|
177
|
+
- Tool performance analysis
|
|
178
|
+
|
|
179
|
+
3. **Individual Reports** (`individual-repos/*/README.md`)
|
|
180
|
+
- Repository-specific findings
|
|
181
|
+
- Tool-by-tool breakdown
|
|
182
|
+
- Detailed issue listings
|
|
183
|
+
- Severity classifications
|
|
184
|
+
|
|
185
|
+
4. **Tool Comparison** (`tool-comparisons/comparison.md`)
|
|
186
|
+
- Detection metrics
|
|
187
|
+
- Tool capabilities matrix
|
|
188
|
+
- Implementation strategy guide
|
|
189
|
+
- Tool selection recommendations
|
|
190
|
+
|
|
191
|
+
### How we normalize findings
|
|
192
|
+
|
|
193
|
+
All tool outputs are converted into a single CommonFinding schema during aggregation. This enables a unified view (JSON/YAML/HTML/SARIF) and consistent gating.
|
|
194
|
+
|
|
195
|
+
- Schema: [docs/schemas/common_finding.v1.json](docs/schemas/common_finding.v1.json)
|
|
196
|
+
- Required fields include: schemaVersion (1.0.0), id, ruleId, severity, tool (name/version), location (path/lines), and message. Optional fields include title, description, remediation, references, tags, cvss, and raw (original tool payload).
|
|
197
|
+
- Fingerprint (id): deterministically derived from a stable subset of attributes (tool | ruleId | path | startLine | message snippet) to support cross-tool dedupe. The aggregation step deduplicates by this id.
|
|
198
|
+
|
|
199
|
+
|
|
200
|
+
|
|
201
|
+
## 🛠️ Tool Installation
|
|
202
|
+
|
|
203
|
+
### macOS (Homebrew)
|
|
204
|
+
|
|
205
|
+
```bash
|
|
206
|
+
# Core tools
|
|
207
|
+
brew install cloc jq
|
|
208
|
+
|
|
209
|
+
# Gitleaks
|
|
210
|
+
brew install gitleaks
|
|
211
|
+
|
|
212
|
+
# Semgrep
|
|
213
|
+
brew install semgrep
|
|
214
|
+
|
|
215
|
+
# TruffleHog
|
|
216
|
+
brew install trufflesecurity/trufflehog/trufflehog
|
|
217
|
+
|
|
218
|
+
# Nosey Parker
|
|
219
|
+
# Download from: https://github.com/praetorian-inc/noseyparker/releases
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
### Linux (Ubuntu/Debian)
|
|
223
|
+
|
|
224
|
+
```bash
|
|
225
|
+
# Core tools
|
|
226
|
+
sudo apt-get install cloc jq
|
|
227
|
+
|
|
228
|
+
# Gitleaks
|
|
229
|
+
wget https://github.com/zricethezav/gitleaks/releases/latest/download/gitleaks-linux-amd64
|
|
230
|
+
chmod +x gitleaks-linux-amd64
|
|
231
|
+
sudo mv gitleaks-linux-amd64 /usr/local/bin/gitleaks
|
|
232
|
+
|
|
233
|
+
# Semgrep
|
|
234
|
+
pip install semgrep
|
|
235
|
+
|
|
236
|
+
# TruffleHog
|
|
237
|
+
curl -sSfL https://raw.githubusercontent.com/trufflesecurity/trufflehog/main/scripts/install.sh | sh -s -- -b /usr/local/bin
|
|
238
|
+
|
|
239
|
+
# Nosey Parker
|
|
240
|
+
# Download from: https://github.com/praetorian-inc/noseyparker/releases
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
### Nosey Parker (manual install)
|
|
244
|
+
|
|
245
|
+
Nosey Parker doesn’t ship via apt/brew universally. Install the release binary and put it on your PATH:
|
|
246
|
+
|
|
247
|
+
1) Download the latest release for your OS/arch from:
|
|
248
|
+
https://github.com/praetorian-inc/noseyparker/releases
|
|
249
|
+
|
|
250
|
+
2) Unpack and move the binary onto PATH (example for Linux x86_64):
|
|
251
|
+
|
|
252
|
+
```bash
|
|
253
|
+
tar -xzf noseyparker-*.tar.gz
|
|
254
|
+
chmod +x noseyparker
|
|
255
|
+
sudo mv noseyparker /usr/local/bin/
|
|
256
|
+
noseyparker --version
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
Tip: run `make verify-env` to confirm the tool is detected.
|
|
260
|
+
|
|
261
|
+
### Nosey Parker on WSL (native recommended) + Docker fallback
|
|
262
|
+
|
|
263
|
+
On WSL Ubuntu, installing Nosey Parker natively is the most reliable path (prebuilt binaries can hit glibc issues). See “User Guide — Nosey Parker on WSL” for a short build-from-source flow using Rust and Boost. When the local binary is not available or fails to run, the CLI automatically falls back to a Docker-based runner.
|
|
264
|
+
|
|
265
|
+
The CLI automatically falls back to a Docker-based Nosey Parker runner when the local binary is missing or not runnable (common on older WSL/glibc). When enabled via profiles, scans will transparently produce the expected JSON here:
|
|
266
|
+
|
|
267
|
+
```
|
|
268
|
+
results/individual-repos/<repo-name>/noseyparker.json
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
Requirements for the fallback:
|
|
272
|
+
- Docker installed and running
|
|
273
|
+
- Ability to pull or use `ghcr.io/praetorian-inc/noseyparker:latest`
|
|
274
|
+
|
|
275
|
+
Manual usage (optional):
|
|
276
|
+
|
|
277
|
+
```bash
|
|
278
|
+
bash scripts/core/run_noseyparker_docker.sh \
|
|
279
|
+
--repo /path/to/repo \
|
|
280
|
+
--out results/individual-repos/<repo-name>/noseyparker.json
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
This mounts your repository read-only into the container, scans it, and writes a JSON report to the `--out` path. The CLI uses this same script automatically when needed.
|
|
284
|
+
|
|
285
|
+
### Semgrep (latest via official script, optional)
|
|
286
|
+
|
|
287
|
+
If you prefer the bleeding-edge standalone installer maintained by Semgrep:
|
|
288
|
+
|
|
289
|
+
```bash
|
|
290
|
+
curl -sL https://semgrep.dev/install.sh | sh
|
|
291
|
+
|
|
292
|
+
# Ensure ~/.local/bin is on PATH (the installer places semgrep there by default)
|
|
293
|
+
export PATH="$HOME/.local/bin:$PATH"
|
|
294
|
+
semgrep --version
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
Note: we recommend isolating CLI tools via pipx or OS packages for stability. The official installer is a convenient alternative when you need the newest release.
|
|
298
|
+
|
|
299
|
+
## 📋 Advanced Usage
|
|
300
|
+
|
|
301
|
+
### Helper Scripts for Multi-Repo Scanning
|
|
302
|
+
|
|
303
|
+
#### `scripts/populate_targets.sh` - Automated Repository Cloning
|
|
304
|
+
|
|
305
|
+
This helper script streamlines the process of cloning multiple repositories for security scanning, with performance optimizations for WSL environments.
|
|
306
|
+
|
|
307
|
+
**Features:**
|
|
308
|
+
- 🚀 Shallow clones (depth=1) for faster cloning
|
|
309
|
+
- ⚡ Parallel cloning for improved performance
|
|
310
|
+
- 🔄 Unshallow option for secret scanners requiring full history
|
|
311
|
+
- 📝 Reads from repository list file
|
|
312
|
+
|
|
313
|
+
**Usage Examples:**
|
|
314
|
+
|
|
315
|
+
```bash
|
|
316
|
+
# Basic usage with defaults (samples/repos.txt → ~/security-testing)
|
|
317
|
+
./scripts/core/populate_targets.sh
|
|
318
|
+
|
|
319
|
+
# Custom repository list and destination
|
|
320
|
+
./scripts/core/populate_targets.sh --list custom-repos.txt --dest ~/my-test-repos
|
|
321
|
+
|
|
322
|
+
# Full clones with 8 parallel jobs
|
|
323
|
+
./scripts/core/populate_targets.sh --full --parallel 8
|
|
324
|
+
|
|
325
|
+
# Unshallow existing shallow clones
|
|
326
|
+
./scripts/core/populate_targets.sh --dest ~/security-testing --unshallow
|
|
327
|
+
|
|
328
|
+
# Show all options
|
|
329
|
+
./scripts/core/populate_targets.sh --help
|
|
330
|
+
```
|
|
331
|
+
|
|
332
|
+
**Repository List Format (`samples/repos.txt`):**
|
|
333
|
+
```
|
|
334
|
+
# One GitHub repository URL per line
|
|
335
|
+
# Lines starting with # are comments
|
|
336
|
+
https://github.com/user/repo1.git
|
|
337
|
+
https://github.com/user/repo2.git
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
**Performance Tips for WSL:**
|
|
341
|
+
1. Use shallow clones initially for 10x faster cloning
|
|
342
|
+
2. Adjust `--parallel` based on network speed (default: 4)
|
|
343
|
+
3. Use `--unshallow` only if secret scanners need full git history
|
|
344
|
+
4. Clone to WSL filesystem (not Windows mount) for better performance
|
|
345
|
+
|
|
346
|
+
### Running Individual Scripts
|
|
347
|
+
|
|
348
|
+
1. **Tool Check Only**:
|
|
349
|
+
```bash
|
|
350
|
+
./scripts/core/check_tools.sh
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
2. **Main Audit Script**:
|
|
354
|
+
```bash
|
|
355
|
+
./scripts/core/run_security_audit.sh [testing_directory] [output_directory]
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
3. **Generate Dashboard Only**:
|
|
359
|
+
```bash
|
|
360
|
+
# Generate dashboard with default output (results_dir/dashboard.html)
|
|
361
|
+
python3 scripts/core/generate_dashboard.py /path/to/results
|
|
362
|
+
|
|
363
|
+
# Generate dashboard with custom output path
|
|
364
|
+
python3 scripts/core/generate_dashboard.py /path/to/results /custom/path/dashboard.html
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
The dashboard generator supports:
|
|
368
|
+
- Multiple TruffleHog output formats (JSON arrays, NDJSON, single objects, empty files)
|
|
369
|
+
- Automatic parent directory creation for custom output paths
|
|
370
|
+
- Graceful handling of missing or empty scan results
|
|
371
|
+
- UTF-8 safe file I/O for international characters
|
|
372
|
+
|
|
373
|
+
### Unified CLI: report-only
|
|
374
|
+
|
|
375
|
+
After scans complete, you can generate unified, normalized reports via the Python CLI:
|
|
376
|
+
|
|
377
|
+
```bash
|
|
378
|
+
# Default reports (formats controlled by jmo.yml)
|
|
379
|
+
python3 scripts/cli/jmo.py report /path/to/security-results
|
|
380
|
+
|
|
381
|
+
# Set thread workers explicitly for aggregation
|
|
382
|
+
python3 scripts/cli/jmo.py report /path/to/security-results --threads 6
|
|
383
|
+
|
|
384
|
+
# Record profiling timings (writes summaries/timings.json)
|
|
385
|
+
python3 scripts/cli/jmo.py report /path/to/security-results --profile
|
|
386
|
+
|
|
387
|
+
# Human-friendly colored logs (stderr)
|
|
388
|
+
python3 scripts/cli/jmo.py report /path/to/security-results --human-logs
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
Or using Make:
|
|
392
|
+
|
|
393
|
+
```bash
|
|
394
|
+
make report RESULTS_DIR=/path/to/security-results THREADS=6
|
|
395
|
+
make profile RESULTS_DIR=/path/to/security-results THREADS=6
|
|
396
|
+
```
|
|
397
|
+
|
|
398
|
+
When profiling is enabled, `timings.json` will include aggregate time, a recommended thread count, and per-job timings.
|
|
399
|
+
|
|
400
|
+
### Unified CLI: scan/ci
|
|
401
|
+
|
|
402
|
+
```bash
|
|
403
|
+
# Scan a single repo with a custom tool subset and timeouts
|
|
404
|
+
python3 scripts/cli/jmo.py scan --repo /path/to/repo --tools gitleaks semgrep --timeout 300 --human-logs
|
|
405
|
+
|
|
406
|
+
# CI convenience – scan then report with gating on severity
|
|
407
|
+
python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name balanced --fail-on HIGH --profile
|
|
408
|
+
```
|
|
409
|
+
|
|
410
|
+
### Output Structure (Summaries)
|
|
411
|
+
|
|
412
|
+
The `summaries/` folder also contains unified outputs:
|
|
413
|
+
|
|
414
|
+
```
|
|
415
|
+
summaries/
|
|
416
|
+
├── findings.json # Unified normalized findings (machine-readable)
|
|
417
|
+
├── SUMMARY.md # Human-readable summary
|
|
418
|
+
├── findings.yaml # Optional YAML (requires PyYAML)
|
|
419
|
+
├── dashboard.html # Self-contained HTML view
|
|
420
|
+
├── findings.sarif # SARIF 2.1.0 for code scanning
|
|
421
|
+
├── SUPPRESSIONS.md # Suppression summary
|
|
422
|
+
└── timings.json # Profiling (when --profile used)
|
|
423
|
+
```
|
|
424
|
+
|
|
425
|
+
### Profiles, per-tool overrides, retries
|
|
426
|
+
|
|
427
|
+
You can define named profiles in `jmo.yml` to control which tools run, include/exclude repo patterns, timeouts, and threads. You can also provide per-tool flags and timeouts, and a global retry count for flaky tools.
|
|
428
|
+
|
|
429
|
+
Example `jmo.yml` snippet:
|
|
430
|
+
|
|
431
|
+
```
|
|
432
|
+
default_profile: fast
|
|
433
|
+
retries: 1
|
|
434
|
+
profiles:
|
|
435
|
+
fast:
|
|
436
|
+
tools: [gitleaks, semgrep]
|
|
437
|
+
include: ["*"]
|
|
438
|
+
exclude: ["big-monorepo*"]
|
|
439
|
+
timeout: 300
|
|
440
|
+
threads: 8
|
|
441
|
+
per_tool:
|
|
442
|
+
semgrep:
|
|
443
|
+
flags: ["--exclude", "node_modules", "--exclude", "dist"]
|
|
444
|
+
timeout: 180
|
|
445
|
+
deep:
|
|
446
|
+
tools: [gitleaks, semgrep, syft, trivy, hadolint, checkov, tfsec, noseyparker]
|
|
447
|
+
timeout: 1200
|
|
448
|
+
threads: 4
|
|
449
|
+
|
|
450
|
+
per_tool:
|
|
451
|
+
trivy:
|
|
452
|
+
flags: ["--ignore-unfixed"]
|
|
453
|
+
timeout: 1200
|
|
454
|
+
```
|
|
455
|
+
|
|
456
|
+
Using a profile from CLI:
|
|
457
|
+
|
|
458
|
+
```
|
|
459
|
+
# Scan using profile 'fast' with human-friendly logs
|
|
460
|
+
python3 scripts/cli/jmo.py scan --repos-dir ~/repos --profile-name fast --human-logs
|
|
461
|
+
|
|
462
|
+
# CI convenience: scan then report, failing on HIGH or worse, record timings, use 'deep' profile
|
|
463
|
+
python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name deep --fail-on HIGH --profile
|
|
464
|
+
```
|
|
465
|
+
|
|
466
|
+
Retries behavior:
|
|
467
|
+
- Global `retries` (or per-profile) retries failed tool commands a limited number of times
|
|
468
|
+
- Some tools use non-zero exit to indicate “findings”; we treat those as success codes to avoid useless retries
|
|
469
|
+
|
|
470
|
+
Human logs show per-tool retry attempts when > 1, e.g.: `attempts={'semgrep': 2}`
|
|
471
|
+
|
|
472
|
+
4. **Generate Comparison Report**:
|
|
473
|
+
```bash
|
|
474
|
+
./scripts/core/generate_comparison_report.sh /path/to/results
|
|
475
|
+
```
|
|
476
|
+
|
|
477
|
+
### Customizing Tool Execution
|
|
478
|
+
|
|
479
|
+
Prefer jmo.yml profiles and per_tool overrides. For one-off local tweaks, use:
|
|
480
|
+
|
|
481
|
+
```bash
|
|
482
|
+
python3 scripts/cli/jmo.py scan --repos-dir ~/repos --tools gitleaks semgrep --timeout 300
|
|
483
|
+
```
|
|
484
|
+
|
|
485
|
+
## 📚 Examples, Screenshots, and Testing
|
|
486
|
+
|
|
487
|
+
- Examples: see `docs/examples/README.md` for common CLI patterns and CI gating.
|
|
488
|
+
- Screenshots: `docs/screenshots/README.md` and `docs/screenshots/capture.sh` to generate dashboard visuals.
|
|
489
|
+
- Testing: see `TEST.md` for running lint, tests, and coverage locally (CI gate ≥85%).
|
|
490
|
+
|
|
491
|
+
## 🔍 Understanding Results
|
|
492
|
+
|
|
493
|
+
### Severity Levels
|
|
494
|
+
|
|
495
|
+
- **CRITICAL**: Verified secrets requiring immediate action
|
|
496
|
+
- **HIGH**: Likely secrets or serious vulnerabilities
|
|
497
|
+
- **MEDIUM**: Potential issues requiring review
|
|
498
|
+
- **LOW/INFO**: Informational findings
|
|
499
|
+
|
|
500
|
+
### Key Metrics
|
|
501
|
+
|
|
502
|
+
- **Total Findings**: All security issues detected
|
|
503
|
+
- **Verified Secrets**: Confirmed active credentials (TruffleHog)
|
|
504
|
+
- **Unique Issues**: Distinct types of security problems
|
|
505
|
+
- **Tool Coverage**: Number of tools that found issues
|
|
506
|
+
|
|
507
|
+
### Recommendations Priority
|
|
508
|
+
|
|
509
|
+
1. **Immediate**: Rotate/revoke verified secrets
|
|
510
|
+
2. **High Priority**: Fix critical and high severity issues
|
|
511
|
+
3. **Medium Priority**: Address medium severity findings
|
|
512
|
+
4. **Long-term**: Implement preventive measures
|
|
513
|
+
|
|
514
|
+
## 🎯 Three-Stage Implementation Strategy
|
|
515
|
+
|
|
516
|
+
### Stage 1: Pre-commit Hooks
|
|
517
|
+
- **Tool**: Gitleaks
|
|
518
|
+
- **Purpose**: Prevent secrets before commit
|
|
519
|
+
- **Speed**: Fast (suitable for developer workflow)
|
|
520
|
+
|
|
521
|
+
### Stage 2: CI/CD Pipeline
|
|
522
|
+
- **Tools**: Gitleaks + Semgrep
|
|
523
|
+
- **Purpose**: Automated PR/commit scanning
|
|
524
|
+
- **Coverage**: Secrets + vulnerabilities
|
|
525
|
+
|
|
526
|
+
### Stage 3: Deep Periodic Audits
|
|
527
|
+
- **Tools**: All tools
|
|
528
|
+
- **Purpose**: Comprehensive security assessment
|
|
529
|
+
- **Frequency**: Weekly/monthly
|
|
530
|
+
|
|
531
|
+
## 📊 Sample Output
|
|
532
|
+
|
|
533
|
+
### Dashboard Preview
|
|
534
|
+
The HTML dashboard provides:
|
|
535
|
+
- Visual metric cards with key statistics
|
|
536
|
+
- Severity breakdown tables
|
|
537
|
+
- Repository-by-repository comparison
|
|
538
|
+
- Tool performance analysis
|
|
539
|
+
- Actionable recommendations
|
|
540
|
+
|
|
541
|
+
### Sample Summary Report
|
|
542
|
+
```markdown
|
|
543
|
+
## Aggregate Results
|
|
544
|
+
|
|
545
|
+
### Overall Statistics
|
|
546
|
+
- Total Issues Found: 1562
|
|
547
|
+
- Critical Issues: 5
|
|
548
|
+
- High Severity Issues: 579
|
|
549
|
+
- Medium Severity Issues: 61
|
|
550
|
+
- Verified Secrets: 5
|
|
551
|
+
|
|
552
|
+
### Recommendations
|
|
553
|
+
- Rotate all 5 verified secrets immediately
|
|
554
|
+
- Prioritize remediation of 584 critical and high severity issues
|
|
555
|
+
- Schedule follow-up review for the remaining medium findings
|
|
556
|
+
```
|
|
557
|
+
|
|
558
|
+
## 🤝 Contributing
|
|
559
|
+
|
|
560
|
+
Contributions are welcome! Please feel free to submit pull requests or open issues for bugs and feature requests.
|
|
561
|
+
|
|
562
|
+
## 📝 License
|
|
563
|
+
|
|
564
|
+
MIT License. See LICENSE.
|
|
565
|
+
|
|
566
|
+
## 🔗 Related Resources
|
|
567
|
+
|
|
568
|
+
- [Gitleaks Documentation](https://github.com/zricethezav/gitleaks)
|
|
569
|
+
- [TruffleHog Documentation](https://github.com/trufflesecurity/trufflehog)
|
|
570
|
+
- [Semgrep Documentation](https://semgrep.dev)
|
|
571
|
+
- [Nosey Parker Documentation](https://github.com/praetorian-inc/noseyparker)
|
|
572
|
+
|
|
573
|
+
## 💡 Tips
|
|
574
|
+
|
|
575
|
+
1. **Start Small**: Test on a single repository first
|
|
576
|
+
2. **Review Regularly**: Schedule periodic audits
|
|
577
|
+
3. **Act Quickly**: Rotate verified secrets immediately
|
|
578
|
+
4. **Prevent Issues**: Implement pre-commit hooks
|
|
579
|
+
5. **Monitor Trends**: Track metrics over time
|
|
580
|
+
|
|
581
|
+
## 🆘 Troubleshooting
|
|
582
|
+
|
|
583
|
+
### Common Issues
|
|
584
|
+
|
|
585
|
+
**Problem**: Tools not found
|
|
586
|
+
- **Solution**: Run `./scripts/cli/security_audit.sh --check` to verify installation
|
|
587
|
+
|
|
588
|
+
**Problem**: JSON parsing errors
|
|
589
|
+
- **Solution**: Ensure jq is installed and tools are outputting valid JSON
|
|
590
|
+
|
|
591
|
+
**Problem**: Permission denied
|
|
592
|
+
- **Solution**: Run `chmod +x *.sh` to make scripts executable
|
|
593
|
+
|
|
594
|
+
**Problem**: Out of memory
|
|
595
|
+
- **Solution**: Scan repositories in smaller batches
|
|
596
|
+
|
|
597
|
+
**Problem**: Path errors (e.g., "//run_security_audit.sh not found")
|
|
598
|
+
- **Solution**: This issue has been fixed in the latest version. Update to the latest main branch.
|
|
599
|
+
- The wrapper scripts now use absolute paths computed from the script's real path location.
|
|
600
|
+
|
|
601
|
+
**Problem**: AttributeError when generating dashboard with TruffleHog results
|
|
602
|
+
- **Solution**: This has been fixed. The dashboard generator now handles all TruffleHog output formats:
|
|
603
|
+
- JSON arrays: `[{...}, {...}]`
|
|
604
|
+
- Single objects: `{...}`
|
|
605
|
+
- NDJSON (one object per line)
|
|
606
|
+
- Empty files or missing files
|
|
607
|
+
- Nested arrays
|
|
608
|
+
|
|
609
|
+
### Rebuilding Reports Without Re-Scanning
|
|
610
|
+
|
|
611
|
+
You can regenerate the dashboard or reports from existing scan results without re-running the security tools:
|
|
612
|
+
|
|
613
|
+
```bash
|
|
614
|
+
# Generate dashboard with default output location
|
|
615
|
+
python3 scripts/core/generate_dashboard.py /path/to/results
|
|
616
|
+
|
|
617
|
+
# Generate dashboard with custom output path (creates parent directories automatically)
|
|
618
|
+
python3 scripts/core/generate_dashboard.py /path/to/results /custom/path/dashboard.html
|
|
619
|
+
|
|
620
|
+
# Example: Generate dashboard in a reports directory
|
|
621
|
+
python3 scripts/core/generate_dashboard.py ~/security-results-20251010-120000 ~/reports/security-dashboard.html
|
|
622
|
+
```
|
|
623
|
+
|
|
624
|
+
This is useful when you want to:
|
|
625
|
+
- Update the dashboard after manually editing JSON files
|
|
626
|
+
- Generate multiple dashboards with different configurations
|
|
627
|
+
- Share results by exporting to a specific location
|
|
628
|
+
|
|
629
|
+
---
|
|
630
|
+
|
|
631
|
+
**Last Updated**: October 10th, 2025
|
|
632
|
+
**Author**: James Moceri
|