jmo-security 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of jmo-security might be problematic. Click here for more details.

@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2025 jimmy058910
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,632 @@
1
+ Metadata-Version: 2.1
2
+ Name: jmo-security
3
+ Version: 0.3.0
4
+ Summary: JMo Security Audit Suite (terminal-first, multi-tool, unified outputs)
5
+ Author-email: James Moceri <general@jmogaming.com>
6
+ License: MIT
7
+ Classifier: Programming Language :: Python :: 3
8
+ Classifier: License :: OSI Approved :: MIT License
9
+ Classifier: Operating System :: OS Independent
10
+ Requires-Python: >=3.8
11
+ Description-Content-Type: text/markdown
12
+ License-File: LICENSE
13
+ Provides-Extra: reporting
14
+ Requires-Dist: PyYAML>=6.0; extra == "reporting"
15
+ Requires-Dist: jsonschema>=4.0; extra == "reporting"
16
+
17
+ # JMo's Security Audit Tool Suite
18
+
19
+ <p align="center">
20
+ <img src="assets/jmo-logo.png" alt="JMo Security Audit Tool Suite" width="220" />
21
+ </p>
22
+
23
+ [![Tests](https://github.com/jimmy058910/jmo-security-repo/actions/workflows/tests.yml/badge.svg)](https://github.com/jimmy058910/jmo-security-repo/actions/workflows/tests.yml)
24
+ [![codecov](https://codecov.io/gh/jimmy058910/jmo-security-repo/branch/main/graph/badge.svg)](https://codecov.io/gh/jimmy058910/jmo-security-repo)
25
+ [![PyPI - Version](https://img.shields.io/pypi/v/jmo-security)](https://pypi.org/project/jmo-security/)
26
+
27
+ A terminal-first, cross-platform security audit toolkit that orchestrates multiple scanners (secrets, SAST, SBOM, IaC, Dockerfile) with a unified Python CLI, normalized outputs, and an HTML dashboard.
28
+
29
+ 👉 New here? Read the comprehensive User Guide: [docs/USER_GUIDE.md](docs/USER_GUIDE.md)
30
+ Docs hub: [docs/index.md](docs/index.md)
31
+
32
+ ## 🎯 Overview
33
+
34
+ This project provides an automated framework for conducting thorough security audits on code repositories. It orchestrates multiple industry-standard security tools to detect secrets, vulnerabilities, and security issues.
35
+
36
+ ### Key Features
37
+
38
+ - ✅ **Multi-Tool Scanning**: Curated set covering secrets (gitleaks, noseyparker), SAST (semgrep, bandit), SBOM+vuln/misconfig (syft+trivy), IaC (checkov, tfsec), Dockerfile (hadolint)
39
+ - 📊 **Comprehensive Reporting**: Unified findings (JSON/YAML), SARIF, Markdown summary, and an interactive HTML dashboard
40
+ - 🎨 **Easy-to-Read Outputs**: Well-formatted reports with severity categorization
41
+ - 🔄 **Automated Workflows**: One CLI to scan, aggregate, and gate on severity (scan/report/ci)
42
+ - 🧭 **Profiles and Overrides**: Named profiles, per-tool flags/timeouts, include/exclude patterns
43
+ - 🔁 **Resilience**: Timeouts, retries with per-tool success codes, human-friendly logs, graceful cancel
44
+
45
+ ## 🚀 Quick Start
46
+
47
+ ### Install or Update (curated tools)
48
+
49
+ These targets detect Linux/WSL/macOS and install or upgrade the curated CLI tools used by this suite. They also surface helpful hints if a platform step needs manual action.
50
+
51
+ ```bash
52
+ make tools # one-time install of curated tools
53
+ make tools-upgrade # refresh/upgrade curated tools
54
+ make verify-env # check OS/WSL/macOS and tool availability
55
+ make dev-deps # install Python dev dependencies
56
+ ```
57
+
58
+ ### Quick Start (Unified CLI)
59
+
60
+ 1) Verify your environment (Linux/WSL/macOS) and see install hints for optional tools:
61
+
62
+ ```bash
63
+ make verify-env
64
+ ```
65
+
66
+ 2) Install Python dev dependencies (for running tests and reporters):
67
+
68
+ ```bash
69
+ make dev-deps
70
+ ```
71
+
72
+ 3) Scan repositories using a profile, then aggregate reports:
73
+
74
+ ```bash
75
+ # Scan immediate subfolders under ~/repos with the 'balanced' profile (default)
76
+ python3 scripts/cli/jmo.py scan --repos-dir ~/repos --profile-name balanced --human-logs
77
+
78
+ # Aggregate and write unified outputs to results/summaries
79
+ python3 scripts/cli/jmo.py report ./results --profile --human-logs
80
+
81
+ # Or do both in one step for CI with a failure threshold
82
+ python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name fast --fail-on HIGH --profile --human-logs
83
+ ```
84
+
85
+ Outputs include: summaries/findings.json, SUMMARY.md, findings.yaml, findings.sarif (enabled by default), dashboard.html, and timings.json (when profiling).
86
+
87
+ ### Basic Usage
88
+
89
+ #### Optional: Quick Setup with Helper Script
90
+
91
+ Use the `populate_targets.sh` helper script to clone multiple repositories for testing (optimized for WSL):
92
+
93
+ ```bash
94
+ # Clone sample vulnerable repos (fast shallow clones)
95
+ ./scripts/core/populate_targets.sh
96
+
97
+ # Clone from custom list with full history
98
+ ./scripts/core/populate_targets.sh --list my-repos.txt --full
99
+
100
+ # Clone with 8 parallel jobs for faster performance
101
+ ./scripts/core/populate_targets.sh --parallel 8
102
+
103
+ # Unshallow repos if secret scanners need full git history
104
+ ./scripts/core/populate_targets.sh --unshallow
105
+ ```
106
+
107
+ #### Running Security Scans (legacy shell script)
108
+
109
+ Prefer the Python CLI above. For legacy flows, you can still use the shell wrapper:
110
+
111
+ ```bash
112
+ ./scripts/cli/security_audit.sh -d ~/security-testing # scan
113
+ ./scripts/cli/security_audit.sh --check # verify tools
114
+ ```
115
+
116
+ #### End-to-End Workflow
117
+
118
+ ```bash
119
+ # 1. Clone test repositories (shallow for speed)
120
+ ./scripts/core/populate_targets.sh --dest ~/test-repos --parallel 4
121
+
122
+ # 2. Run security audit (preferred)
123
+ python3 scripts/cli/jmo.py ci --repos-dir ~/test-repos --fail-on HIGH --profile --human-logs
124
+
125
+ # 3. View results
126
+ cat ~/security-results-*/SUMMARY_REPORT.md
127
+ # macOS: open ~/security-results-*/dashboard.html
128
+ # Linux: xdg-open ~/security-results-*/dashboard.html
129
+ ```
130
+
131
+ ## 📚 Documentation
132
+
133
+ ### Workflow (at a glance)
134
+
135
+ The security audit follows this workflow:
136
+
137
+ 1. **Tool Verification**: Checks all required tools are installed
138
+ 2. **Repository Scanning**: jmo scan orchestrates tools per jmo.yml (profiles, overrides, retries)
139
+ 3. **Results Aggregation**: jmo report normalizes tool outputs to a CommonFinding shape
140
+ 4. **Report Generation**: JSON/MD/YAML/HTML/SARIF and suppression summary
141
+ 5. **Dashboard Creation**: Self-contained HTML dashboard with an optional profiling panel
142
+
143
+ ### Output Structure
144
+
145
+ ```
146
+ security-results-YYYYMMDD-HHMMSS/
147
+ ├── SUMMARY_REPORT.md # Executive summary
148
+ ├── dashboard.html # Interactive HTML dashboard
149
+ ├── individual-repos/ # Per-repository results
150
+ │ └── [repo-name]/
151
+ │ ├── README.md # Formatted findings report
152
+ │ ├── gitleaks.json # Gitleaks raw output
153
+ │ ├── trufflehog.json # TruffleHog raw output
154
+ │ ├── semgrep.json # Semgrep raw output
155
+ │ ├── noseyparker.json # Nosey Parker raw output
156
+ │ └── *.log # Tool execution logs
157
+ ├── raw-outputs/ # Compressed per-repo raw artifacts
158
+ ├── tool-comparisons/
159
+ │ └── comparison.md # Tool performance comparison
160
+ ├── summaries/
161
+ │ └── metrics.csv # Aggregated metrics
162
+
163
+ ```
164
+
165
+ ### Report Types
166
+
167
+ 1. **Summary Report** (`SUMMARY_REPORT.md`)
168
+ - Executive summary
169
+ - Aggregate statistics
170
+ - Repository breakdown table
171
+ - Prioritized recommendations
172
+
173
+ 2. **HTML Dashboard** (`dashboard.html`)
174
+ - Visual metrics cards
175
+ - Severity breakdown
176
+ - Repository comparison table
177
+ - Tool performance analysis
178
+
179
+ 3. **Individual Reports** (`individual-repos/*/README.md`)
180
+ - Repository-specific findings
181
+ - Tool-by-tool breakdown
182
+ - Detailed issue listings
183
+ - Severity classifications
184
+
185
+ 4. **Tool Comparison** (`tool-comparisons/comparison.md`)
186
+ - Detection metrics
187
+ - Tool capabilities matrix
188
+ - Implementation strategy guide
189
+ - Tool selection recommendations
190
+
191
+ ### How we normalize findings
192
+
193
+ All tool outputs are converted into a single CommonFinding schema during aggregation. This enables a unified view (JSON/YAML/HTML/SARIF) and consistent gating.
194
+
195
+ - Schema: [docs/schemas/common_finding.v1.json](docs/schemas/common_finding.v1.json)
196
+ - Required fields include: schemaVersion (1.0.0), id, ruleId, severity, tool (name/version), location (path/lines), and message. Optional fields include title, description, remediation, references, tags, cvss, and raw (original tool payload).
197
+ - Fingerprint (id): deterministically derived from a stable subset of attributes (tool | ruleId | path | startLine | message snippet) to support cross-tool dedupe. The aggregation step deduplicates by this id.
198
+
199
+
200
+
201
+ ## 🛠️ Tool Installation
202
+
203
+ ### macOS (Homebrew)
204
+
205
+ ```bash
206
+ # Core tools
207
+ brew install cloc jq
208
+
209
+ # Gitleaks
210
+ brew install gitleaks
211
+
212
+ # Semgrep
213
+ brew install semgrep
214
+
215
+ # TruffleHog
216
+ brew install trufflesecurity/trufflehog/trufflehog
217
+
218
+ # Nosey Parker
219
+ # Download from: https://github.com/praetorian-inc/noseyparker/releases
220
+ ```
221
+
222
+ ### Linux (Ubuntu/Debian)
223
+
224
+ ```bash
225
+ # Core tools
226
+ sudo apt-get install cloc jq
227
+
228
+ # Gitleaks
229
+ wget https://github.com/zricethezav/gitleaks/releases/latest/download/gitleaks-linux-amd64
230
+ chmod +x gitleaks-linux-amd64
231
+ sudo mv gitleaks-linux-amd64 /usr/local/bin/gitleaks
232
+
233
+ # Semgrep
234
+ pip install semgrep
235
+
236
+ # TruffleHog
237
+ curl -sSfL https://raw.githubusercontent.com/trufflesecurity/trufflehog/main/scripts/install.sh | sh -s -- -b /usr/local/bin
238
+
239
+ # Nosey Parker
240
+ # Download from: https://github.com/praetorian-inc/noseyparker/releases
241
+ ```
242
+
243
+ ### Nosey Parker (manual install)
244
+
245
+ Nosey Parker doesn’t ship via apt/brew universally. Install the release binary and put it on your PATH:
246
+
247
+ 1) Download the latest release for your OS/arch from:
248
+ https://github.com/praetorian-inc/noseyparker/releases
249
+
250
+ 2) Unpack and move the binary onto PATH (example for Linux x86_64):
251
+
252
+ ```bash
253
+ tar -xzf noseyparker-*.tar.gz
254
+ chmod +x noseyparker
255
+ sudo mv noseyparker /usr/local/bin/
256
+ noseyparker --version
257
+ ```
258
+
259
+ Tip: run `make verify-env` to confirm the tool is detected.
260
+
261
+ ### Nosey Parker on WSL (native recommended) + Docker fallback
262
+
263
+ On WSL Ubuntu, installing Nosey Parker natively is the most reliable path (prebuilt binaries can hit glibc issues). See “User Guide — Nosey Parker on WSL” for a short build-from-source flow using Rust and Boost. When the local binary is not available or fails to run, the CLI automatically falls back to a Docker-based runner.
264
+
265
+ The CLI automatically falls back to a Docker-based Nosey Parker runner when the local binary is missing or not runnable (common on older WSL/glibc). When enabled via profiles, scans will transparently produce the expected JSON here:
266
+
267
+ ```
268
+ results/individual-repos/<repo-name>/noseyparker.json
269
+ ```
270
+
271
+ Requirements for the fallback:
272
+ - Docker installed and running
273
+ - Ability to pull or use `ghcr.io/praetorian-inc/noseyparker:latest`
274
+
275
+ Manual usage (optional):
276
+
277
+ ```bash
278
+ bash scripts/core/run_noseyparker_docker.sh \
279
+ --repo /path/to/repo \
280
+ --out results/individual-repos/<repo-name>/noseyparker.json
281
+ ```
282
+
283
+ This mounts your repository read-only into the container, scans it, and writes a JSON report to the `--out` path. The CLI uses this same script automatically when needed.
284
+
285
+ ### Semgrep (latest via official script, optional)
286
+
287
+ If you prefer the bleeding-edge standalone installer maintained by Semgrep:
288
+
289
+ ```bash
290
+ curl -sL https://semgrep.dev/install.sh | sh
291
+
292
+ # Ensure ~/.local/bin is on PATH (the installer places semgrep there by default)
293
+ export PATH="$HOME/.local/bin:$PATH"
294
+ semgrep --version
295
+ ```
296
+
297
+ Note: we recommend isolating CLI tools via pipx or OS packages for stability. The official installer is a convenient alternative when you need the newest release.
298
+
299
+ ## 📋 Advanced Usage
300
+
301
+ ### Helper Scripts for Multi-Repo Scanning
302
+
303
+ #### `scripts/populate_targets.sh` - Automated Repository Cloning
304
+
305
+ This helper script streamlines the process of cloning multiple repositories for security scanning, with performance optimizations for WSL environments.
306
+
307
+ **Features:**
308
+ - 🚀 Shallow clones (depth=1) for faster cloning
309
+ - ⚡ Parallel cloning for improved performance
310
+ - 🔄 Unshallow option for secret scanners requiring full history
311
+ - 📝 Reads from repository list file
312
+
313
+ **Usage Examples:**
314
+
315
+ ```bash
316
+ # Basic usage with defaults (samples/repos.txt → ~/security-testing)
317
+ ./scripts/core/populate_targets.sh
318
+
319
+ # Custom repository list and destination
320
+ ./scripts/core/populate_targets.sh --list custom-repos.txt --dest ~/my-test-repos
321
+
322
+ # Full clones with 8 parallel jobs
323
+ ./scripts/core/populate_targets.sh --full --parallel 8
324
+
325
+ # Unshallow existing shallow clones
326
+ ./scripts/core/populate_targets.sh --dest ~/security-testing --unshallow
327
+
328
+ # Show all options
329
+ ./scripts/core/populate_targets.sh --help
330
+ ```
331
+
332
+ **Repository List Format (`samples/repos.txt`):**
333
+ ```
334
+ # One GitHub repository URL per line
335
+ # Lines starting with # are comments
336
+ https://github.com/user/repo1.git
337
+ https://github.com/user/repo2.git
338
+ ```
339
+
340
+ **Performance Tips for WSL:**
341
+ 1. Use shallow clones initially for 10x faster cloning
342
+ 2. Adjust `--parallel` based on network speed (default: 4)
343
+ 3. Use `--unshallow` only if secret scanners need full git history
344
+ 4. Clone to WSL filesystem (not Windows mount) for better performance
345
+
346
+ ### Running Individual Scripts
347
+
348
+ 1. **Tool Check Only**:
349
+ ```bash
350
+ ./scripts/core/check_tools.sh
351
+ ```
352
+
353
+ 2. **Main Audit Script**:
354
+ ```bash
355
+ ./scripts/core/run_security_audit.sh [testing_directory] [output_directory]
356
+ ```
357
+
358
+ 3. **Generate Dashboard Only**:
359
+ ```bash
360
+ # Generate dashboard with default output (results_dir/dashboard.html)
361
+ python3 scripts/core/generate_dashboard.py /path/to/results
362
+
363
+ # Generate dashboard with custom output path
364
+ python3 scripts/core/generate_dashboard.py /path/to/results /custom/path/dashboard.html
365
+ ```
366
+
367
+ The dashboard generator supports:
368
+ - Multiple TruffleHog output formats (JSON arrays, NDJSON, single objects, empty files)
369
+ - Automatic parent directory creation for custom output paths
370
+ - Graceful handling of missing or empty scan results
371
+ - UTF-8 safe file I/O for international characters
372
+
373
+ ### Unified CLI: report-only
374
+
375
+ After scans complete, you can generate unified, normalized reports via the Python CLI:
376
+
377
+ ```bash
378
+ # Default reports (formats controlled by jmo.yml)
379
+ python3 scripts/cli/jmo.py report /path/to/security-results
380
+
381
+ # Set thread workers explicitly for aggregation
382
+ python3 scripts/cli/jmo.py report /path/to/security-results --threads 6
383
+
384
+ # Record profiling timings (writes summaries/timings.json)
385
+ python3 scripts/cli/jmo.py report /path/to/security-results --profile
386
+
387
+ # Human-friendly colored logs (stderr)
388
+ python3 scripts/cli/jmo.py report /path/to/security-results --human-logs
389
+ ```
390
+
391
+ Or using Make:
392
+
393
+ ```bash
394
+ make report RESULTS_DIR=/path/to/security-results THREADS=6
395
+ make profile RESULTS_DIR=/path/to/security-results THREADS=6
396
+ ```
397
+
398
+ When profiling is enabled, `timings.json` will include aggregate time, a recommended thread count, and per-job timings.
399
+
400
+ ### Unified CLI: scan/ci
401
+
402
+ ```bash
403
+ # Scan a single repo with a custom tool subset and timeouts
404
+ python3 scripts/cli/jmo.py scan --repo /path/to/repo --tools gitleaks semgrep --timeout 300 --human-logs
405
+
406
+ # CI convenience – scan then report with gating on severity
407
+ python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name balanced --fail-on HIGH --profile
408
+ ```
409
+
410
+ ### Output Structure (Summaries)
411
+
412
+ The `summaries/` folder also contains unified outputs:
413
+
414
+ ```
415
+ summaries/
416
+ ├── findings.json # Unified normalized findings (machine-readable)
417
+ ├── SUMMARY.md # Human-readable summary
418
+ ├── findings.yaml # Optional YAML (requires PyYAML)
419
+ ├── dashboard.html # Self-contained HTML view
420
+ ├── findings.sarif # SARIF 2.1.0 for code scanning
421
+ ├── SUPPRESSIONS.md # Suppression summary
422
+ └── timings.json # Profiling (when --profile used)
423
+ ```
424
+
425
+ ### Profiles, per-tool overrides, retries
426
+
427
+ You can define named profiles in `jmo.yml` to control which tools run, include/exclude repo patterns, timeouts, and threads. You can also provide per-tool flags and timeouts, and a global retry count for flaky tools.
428
+
429
+ Example `jmo.yml` snippet:
430
+
431
+ ```
432
+ default_profile: fast
433
+ retries: 1
434
+ profiles:
435
+ fast:
436
+ tools: [gitleaks, semgrep]
437
+ include: ["*"]
438
+ exclude: ["big-monorepo*"]
439
+ timeout: 300
440
+ threads: 8
441
+ per_tool:
442
+ semgrep:
443
+ flags: ["--exclude", "node_modules", "--exclude", "dist"]
444
+ timeout: 180
445
+ deep:
446
+ tools: [gitleaks, semgrep, syft, trivy, hadolint, checkov, tfsec, noseyparker]
447
+ timeout: 1200
448
+ threads: 4
449
+
450
+ per_tool:
451
+ trivy:
452
+ flags: ["--ignore-unfixed"]
453
+ timeout: 1200
454
+ ```
455
+
456
+ Using a profile from CLI:
457
+
458
+ ```
459
+ # Scan using profile 'fast' with human-friendly logs
460
+ python3 scripts/cli/jmo.py scan --repos-dir ~/repos --profile-name fast --human-logs
461
+
462
+ # CI convenience: scan then report, failing on HIGH or worse, record timings, use 'deep' profile
463
+ python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name deep --fail-on HIGH --profile
464
+ ```
465
+
466
+ Retries behavior:
467
+ - Global `retries` (or per-profile) retries failed tool commands a limited number of times
468
+ - Some tools use non-zero exit to indicate “findings”; we treat those as success codes to avoid useless retries
469
+
470
+ Human logs show per-tool retry attempts when > 1, e.g.: `attempts={'semgrep': 2}`
471
+
472
+ 4. **Generate Comparison Report**:
473
+ ```bash
474
+ ./scripts/core/generate_comparison_report.sh /path/to/results
475
+ ```
476
+
477
+ ### Customizing Tool Execution
478
+
479
+ Prefer jmo.yml profiles and per_tool overrides. For one-off local tweaks, use:
480
+
481
+ ```bash
482
+ python3 scripts/cli/jmo.py scan --repos-dir ~/repos --tools gitleaks semgrep --timeout 300
483
+ ```
484
+
485
+ ## 📚 Examples, Screenshots, and Testing
486
+
487
+ - Examples: see `docs/examples/README.md` for common CLI patterns and CI gating.
488
+ - Screenshots: `docs/screenshots/README.md` and `docs/screenshots/capture.sh` to generate dashboard visuals.
489
+ - Testing: see `TEST.md` for running lint, tests, and coverage locally (CI gate ≥85%).
490
+
491
+ ## 🔍 Understanding Results
492
+
493
+ ### Severity Levels
494
+
495
+ - **CRITICAL**: Verified secrets requiring immediate action
496
+ - **HIGH**: Likely secrets or serious vulnerabilities
497
+ - **MEDIUM**: Potential issues requiring review
498
+ - **LOW/INFO**: Informational findings
499
+
500
+ ### Key Metrics
501
+
502
+ - **Total Findings**: All security issues detected
503
+ - **Verified Secrets**: Confirmed active credentials (TruffleHog)
504
+ - **Unique Issues**: Distinct types of security problems
505
+ - **Tool Coverage**: Number of tools that found issues
506
+
507
+ ### Recommendations Priority
508
+
509
+ 1. **Immediate**: Rotate/revoke verified secrets
510
+ 2. **High Priority**: Fix critical and high severity issues
511
+ 3. **Medium Priority**: Address medium severity findings
512
+ 4. **Long-term**: Implement preventive measures
513
+
514
+ ## 🎯 Three-Stage Implementation Strategy
515
+
516
+ ### Stage 1: Pre-commit Hooks
517
+ - **Tool**: Gitleaks
518
+ - **Purpose**: Prevent secrets before commit
519
+ - **Speed**: Fast (suitable for developer workflow)
520
+
521
+ ### Stage 2: CI/CD Pipeline
522
+ - **Tools**: Gitleaks + Semgrep
523
+ - **Purpose**: Automated PR/commit scanning
524
+ - **Coverage**: Secrets + vulnerabilities
525
+
526
+ ### Stage 3: Deep Periodic Audits
527
+ - **Tools**: All tools
528
+ - **Purpose**: Comprehensive security assessment
529
+ - **Frequency**: Weekly/monthly
530
+
531
+ ## 📊 Sample Output
532
+
533
+ ### Dashboard Preview
534
+ The HTML dashboard provides:
535
+ - Visual metric cards with key statistics
536
+ - Severity breakdown tables
537
+ - Repository-by-repository comparison
538
+ - Tool performance analysis
539
+ - Actionable recommendations
540
+
541
+ ### Sample Summary Report
542
+ ```markdown
543
+ ## Aggregate Results
544
+
545
+ ### Overall Statistics
546
+ - Total Issues Found: 1562
547
+ - Critical Issues: 5
548
+ - High Severity Issues: 579
549
+ - Medium Severity Issues: 61
550
+ - Verified Secrets: 5
551
+
552
+ ### Recommendations
553
+ - Rotate all 5 verified secrets immediately
554
+ - Prioritize remediation of 584 critical and high severity issues
555
+ - Schedule follow-up review for the remaining medium findings
556
+ ```
557
+
558
+ ## 🤝 Contributing
559
+
560
+ Contributions are welcome! Please feel free to submit pull requests or open issues for bugs and feature requests.
561
+
562
+ ## 📝 License
563
+
564
+ MIT License. See LICENSE.
565
+
566
+ ## 🔗 Related Resources
567
+
568
+ - [Gitleaks Documentation](https://github.com/zricethezav/gitleaks)
569
+ - [TruffleHog Documentation](https://github.com/trufflesecurity/trufflehog)
570
+ - [Semgrep Documentation](https://semgrep.dev)
571
+ - [Nosey Parker Documentation](https://github.com/praetorian-inc/noseyparker)
572
+
573
+ ## 💡 Tips
574
+
575
+ 1. **Start Small**: Test on a single repository first
576
+ 2. **Review Regularly**: Schedule periodic audits
577
+ 3. **Act Quickly**: Rotate verified secrets immediately
578
+ 4. **Prevent Issues**: Implement pre-commit hooks
579
+ 5. **Monitor Trends**: Track metrics over time
580
+
581
+ ## 🆘 Troubleshooting
582
+
583
+ ### Common Issues
584
+
585
+ **Problem**: Tools not found
586
+ - **Solution**: Run `./scripts/cli/security_audit.sh --check` to verify installation
587
+
588
+ **Problem**: JSON parsing errors
589
+ - **Solution**: Ensure jq is installed and tools are outputting valid JSON
590
+
591
+ **Problem**: Permission denied
592
+ - **Solution**: Run `chmod +x *.sh` to make scripts executable
593
+
594
+ **Problem**: Out of memory
595
+ - **Solution**: Scan repositories in smaller batches
596
+
597
+ **Problem**: Path errors (e.g., "//run_security_audit.sh not found")
598
+ - **Solution**: This issue has been fixed in the latest version. Update to the latest main branch.
599
+ - The wrapper scripts now use absolute paths computed from the script's real path location.
600
+
601
+ **Problem**: AttributeError when generating dashboard with TruffleHog results
602
+ - **Solution**: This has been fixed. The dashboard generator now handles all TruffleHog output formats:
603
+ - JSON arrays: `[{...}, {...}]`
604
+ - Single objects: `{...}`
605
+ - NDJSON (one object per line)
606
+ - Empty files or missing files
607
+ - Nested arrays
608
+
609
+ ### Rebuilding Reports Without Re-Scanning
610
+
611
+ You can regenerate the dashboard or reports from existing scan results without re-running the security tools:
612
+
613
+ ```bash
614
+ # Generate dashboard with default output location
615
+ python3 scripts/core/generate_dashboard.py /path/to/results
616
+
617
+ # Generate dashboard with custom output path (creates parent directories automatically)
618
+ python3 scripts/core/generate_dashboard.py /path/to/results /custom/path/dashboard.html
619
+
620
+ # Example: Generate dashboard in a reports directory
621
+ python3 scripts/core/generate_dashboard.py ~/security-results-20251010-120000 ~/reports/security-dashboard.html
622
+ ```
623
+
624
+ This is useful when you want to:
625
+ - Update the dashboard after manually editing JSON files
626
+ - Generate multiple dashboards with different configurations
627
+ - Share results by exporting to a specific location
628
+
629
+ ---
630
+
631
+ **Last Updated**: October 10th, 2025
632
+ **Author**: James Moceri