agentic-threat-hunting-framework 0.2.3__py3-none-any.whl → 0.3.0__py3-none-any.whl

athf/data/hunts/H-0002.md

@@ -0,0 +1,436 @@
+---
+hunt_id: H-0002
+title: Linux Crontab Persistence Detection
+status: completed
+date: 2025-11-19
+hunter: [Your Name]
+platform: [Linux]
+tactics: [persistence]
+techniques: [T1053.003, T1059.004, T1071.001, T1027]
+data_sources: [Auditd, Syslog, Linux Secure Logs]
+related_hunts: []
+findings_count: 3
+true_positives: 1
+false_positives: 1
+customer_deliverables: []
+tags: [linux, cron, persistence, scheduled-tasks, cryptomining]
+---
+
+# H-0002: Linux Crontab Persistence Detection
+
+**Hunt Metadata**
+
+- **Date:** 2025-11-19
+- **Hunter:** [Your Name]
+- **Status:** Completed
+- **MITRE ATT&CK:** T1053.003 - Scheduled Task/Job: Cron
+
+---
+
+## LEARN: Prepare the Hunt
+
+### Hypothesis Statement
+
+Detect adversary persistence via malicious cron job creation or modification (T1053.003) on Linux systems by identifying suspicious crontab entries, file modifications outside maintenance windows, and execution of unusual commands from cron processes.
+
+### Threat Context
+
+Adversaries may abuse cron on Linux systems to establish persistence by scheduling malicious commands or scripts to execute at system startup or on a defined schedule. Cron jobs are commonly used by threat actors to maintain access, execute payloads, or perform reconnaissance activities. This technique allows attackers to survive system reboots and maintain long-term access to compromised hosts.
+
+Key persistence locations:
+
+- User crontabs: `/var/spool/cron/crontabs/*`
+- System-wide: `/etc/crontab`, `/etc/cron.d/*`
+- Scheduled directories: `/etc/cron.hourly/`, `/etc/cron.daily/`, `/etc/cron.weekly/`, `/etc/cron.monthly/`
+
+Common adversary patterns:
+
+- Unusual timing patterns (every minute, odd schedules)
+- Commands with network activity (curl, wget, nc)
+- Obfuscated or base64-encoded commands
+- Execution from temporary directories (/tmp, /dev/shm)
+- Reverse shells or callback mechanisms
+- Crontab modifications by non-administrative users
+
+### ABLE Scoping
+
+Define your hunt scope using the ABLE framework:
+
+| **Field**   | **Your Input** |
+|-------------|----------------|
+| **Actor** *(Optional)* | N/A - Focus on persistence behavior patterns |
+| **Behavior** | Malicious cron job creation/modification for persistence (T1053.003 - Scheduled Task/Job: Cron) |
+| **Location** | All Linux servers (production web servers, database servers, application servers) |
+| **Evidence** | **Source:** Auditd file integrity monitoring<br>**Key Fields:** file_path, action (modified/created), user, process_name, timestamp<br>**Example:** Non-root user modifying /var/spool/cron/crontabs/* with curl/wget commands<br><br>**Source:** Syslog / Linux Secure logs<br>**Key Fields:** parent_process, command_line, user, cron_schedule<br>**Example:** Cron daemon spawning bash with suspicious network commands |
+
+### Threat Intel & Research
+
+- **MITRE ATT&CK Techniques:**
+  - `T1053.003 - Scheduled Task/Job: Cron`
+  - `T1059.004 - Command and Scripting Interpreter: Unix Shell` (often used together)
+  - `T1071.001 - Application Layer Protocol: Web Protocols` (curl/wget for C2)
+  - `T1027 - Obfuscated Files or Information` (base64-encoded cron commands)
+- **CTI Sources & References:**
+  - [MITRE ATT&CK - T1053.003](https://attack.mitre.org/techniques/T1053/003/)
+  - Common in Linux post-exploitation frameworks (Metasploit, Empire, Cobalt Strike)
+  - APT groups using cron persistence: APT28, Rocke (cryptomining), TeamTNT (cloud)
+- **Historical Context:**
+  - Previous incident (2024-Q3): Cryptomining malware used cron for persistence on dev servers
+  - Current baseline: ~200 legitimate cron jobs across production environment
+  - Known false positives: Certbot, log rotation, backup scripts, package managers
+
+### Related Tickets
+
+| **Team** | **Ticket/Details** |
+|----------|-------------------|
+| **SOC/IR** | N/A |
+| **Threat Intel** | TI-0089 - Cryptomining campaign targeting cloud infrastructure |
+| **Detection Engineering** | DET-0051 - Implement file integrity monitoring for cron directories |
+| **Other** | INFRA-1456 - Baseline legitimate cron jobs per host type |
+
+---
+
+## OBSERVE: Expected Behaviors
+
+### What Normal Looks Like
+
+Legitimate cron activity that should not trigger alerts:
+
+- System package managers scheduling updates (apt, yum, dnf) in /etc/cron.daily/
+- Certificate renewal tools (certbot, acme.sh) running in /etc/cron.d/
+- Log rotation via logrotate in /etc/cron.daily/
+- Backup scripts scheduled by root user during maintenance windows (typically 02:00-04:00 UTC)
+- Monitoring agents (Datadog, New Relic) running health checks
+- Database maintenance jobs (vacuum, analyze) scheduled by database users
+- Application-specific tasks managed by service accounts with documented purposes
+
+### What Suspicious Looks Like
+
+Adversaries will modify crontab files to achieve persistence on Linux hosts. We expect to see:
+
+1. **File modifications** to crontab-related files outside normal maintenance windows
+2. **Suspicious commands** in cron entries containing:
+   - Network utilities: `curl`, `wget`, `nc`, `ncat`, `socat`
+   - Shell invocations: `bash -c`, `sh -c`, `/dev/tcp/`
+   - Encoding: `base64`, `echo`, piped to `sh` or `bash`
+   - Temporary paths: `/tmp`, `/dev/shm`, `/var/tmp`
+3. **Unusual crontab users** - modifications by non-root, non-admin accounts
+4. **Process execution** from cron daemon spawning unexpected commands
+5. **New cron files** created in `/etc/cron.d/` with suspicious ownership
+
+### Expected Observables
+
+- **Processes:** Cron daemon spawning bash/sh with unusual command lines, network utilities (curl, wget, nc)
+- **Network:** Outbound connections from cron-spawned processes to external IPs or unusual ports
+- **Files:** Modifications to /etc/crontab, /var/spool/cron/crontabs/*, /etc/cron.d/*, /etc/cron.{hourly,daily,weekly,monthly}/
+- **Registry:** N/A (Linux-based hunt)
+- **Authentication:** Crontab command execution by non-administrative users, unusual process ownership
+
+---
+
+## CHECK: Execute & Analyze
+
+### Data Source Information
+
+- **Index/Data Source:** index=linux, auditd logs, syslog, bash_history
+- **Time Range:** Last 7 days (2025-11-12 00:00:00 to 2025-11-19 23:59:59)
+- **Events Analyzed:** ~50,000 auditd events, ~30,000 syslog entries
+- **Data Quality:** Fair - Auditd deployed on production servers (80% coverage), Dev environments lack FIM monitoring
+
+### Hunting Queries
+
+#### Initial Query: File Integrity Monitoring for Crontab Changes
+
+```bash
+# For hosts with auditd logging
+# Optimized: All filters in base search for maximum efficiency
+index=linux sourcetype=auditd
+file_path IN ("/etc/crontab", "/etc/cron.d/*", "/var/spool/cron/crontabs/*", "/etc/cron.hourly/*", "/etc/cron.daily/*", "/etc/cron.weekly/*", "/etc/cron.monthly/*")
+action IN ("modified", "created", "written")
+| stats count by _time, host, user, file_path, action, process_name
+| where user!="root" OR process_name!="crontab"
+| sort -_time
+```
+
+**Query Notes:**
+
+- Returned 12 crontab modification events over 7-day period
+- 3 events from root user during maintenance window (expected)
+- 2 suspicious events from "webadmin" user modifying /var/spool/cron/crontabs/webadmin
+- 7 events from certbot (legitimate certificate renewal)
+- Most modifications occurred during business hours (09:00-17:00 UTC)
+
+#### Refined Query: Suspicious Cron Command Patterns
+
+```bash
+# For hosts collecting cron file contents
+# Optimized: Filters in base search, pattern matching post-extraction
+index=linux (sourcetype=linux_secure OR sourcetype=syslog)
+("CRON" OR "crontab")
+| rex field=_raw "(?<cron_command>\*.*\s+.+)"
+| search cron_command IN ("*curl*", "*wget*", "*nc *", "*ncat*", "*socat*", "*bash -c*", "*sh -c*", "*base64*", "*/tmp/*", "*/dev/shm/*", "*python -c*", "*perl -e*", "*/dev/tcp/*")
+| stats count by host, cron_command, user
+| sort -count
+```
+
+**Refinement Rationale:**
+
+- Shifted from file modification detection to content-based analysis
+- Added pattern matching for specific malicious command indicators (curl, wget, base64, etc.)
+- Focused on network utilities and obfuscation techniques commonly used for C2
+- Prioritized cron entries executing from temporary directories (/tmp, /dev/shm)
+- This query identifies the actual malicious payload, not just file changes
+
+### Visualization & Analytics
+
+- **Timeline:** Crontab modification events over 7 days show clustering during maintenance windows (expected)
+- **Heatmap:** User vs. file_path shows "webadmin" user modifying personal crontab (requires investigation)
+- **Process tree:** Cron daemon spawning unexpected bash processes with network commands
+- **Command frequency:** 98% of cron commands are legitimate (certbot, logrotate, apt), 2% require investigation
+
+### Query Performance
+
+**What Worked Well:**
+
+- Auditd file integrity monitoring effectively captured crontab modifications with full context
+- Pattern matching on suspicious command strings identified high-risk cron entries
+- User-based filtering (user!="root") reduced noise from legitimate system maintenance
+- Process execution monitoring provided visibility into actual cron command execution
+
+**What Didn't Work:**
+
+- Initial query too broad - captured all crontab modifications including legitimate changes
+- Regex extraction of cron_command field unreliable due to varied log formats
+- Query 3 (process execution) generated false positives from legitimate system processes
+- No baseline of "known good" cron jobs to compare against
+- Bash history logging not consistently enabled across all servers (telemetry gap)
+
+**Iterations Made:**
+
+- Iteration 1: Added user filter to exclude root user modifications (reduced results by 70%)
+- Iteration 2: Created separate query for suspicious command patterns vs. file modifications
+- Iteration 3: Whitelisted common legitimate processes (certbot, logrotate, apt, yum)
+- Iteration 4: Adjusted time range from 24h to 7 days for better pattern visibility
+
+### Manual Validation Steps
+
+```bash
+# On suspected hosts, check current crontabs
+for user in $(cut -f1 -d: /etc/passwd); do
+    echo "=== Crontab for $user ==="
+    crontab -u $user -l 2>/dev/null
+done
+
+# Check system crontabs
+cat /etc/crontab
+ls -la /etc/cron.d/
+ls -la /etc/cron.{hourly,daily,weekly,monthly}/
+
+# Review recent modifications
+find /etc/cron* /var/spool/cron -type f -mtime -7 -ls
+find /etc/cron* /var/spool/cron -type f -exec grep -l "curl\|wget\|nc\|bash -c\|base64\|/tmp" {} \;
+```
+
+---
+
+## KEEP: Findings & Response
+
+### Executive Summary
+
+This hunt investigated cron-based persistence mechanisms (T1053.003) on Linux systems over a 7-day period. Analysis of 50,000 auditd events and 30,000 syslog entries identified 12 crontab modification events. The hypothesis was partially confirmed: while most crontab activity was legitimate system maintenance, one suspicious case was detected requiring investigation. A non-administrative user "webadmin" modified their personal crontab with commands containing network utilities, indicating potential adversary persistence. No confirmed malicious cron jobs were found, but the suspicious activity warrants incident response follow-up. Overall, 98% of cron activity was benign (certbot, logrotate, package managers), with 2% requiring further investigation.
+
+### Findings
+
+| **Finding** | **Ticket** | **Description** |
+|-------------|-----------|-----------------|
+| Suspicious | SOC-2901 | User "webadmin" modified /var/spool/cron/crontabs/webadmin with curl command to external IP - requires IR investigation |
+| True Positive | N/A | Legitimate cron modifications by root during maintenance window (3 events, expected) |
+| True Positive | N/A | Certbot automatic certificate renewal cron entries (7 events, benign) |
+| False Positive | N/A | Database backup scripts using curl to upload to S3 (legitimate but flagged due to curl pattern) |
+
+**True Positives:** 1 suspicious case requiring investigation
+**False Positives:** 1 legitimate backup script using curl
+**Suspicious Events:** 1 requiring immediate incident response investigation (webadmin user)
+
+### Detection Logic
+
+**Automation Opportunity:**
+
+This hunt can be automated with the following approach:
+
+- Alert on crontab file modifications by non-root users
+- Pattern match cron commands for suspicious indicators (curl, wget, nc, base64, /tmp)
+- Baseline legitimate cron jobs per host type to reduce false positives
+- Combine file modification detection with command content analysis
+- Correlate cron activity with network connections from cron-spawned processes
+
+**Proposed Detection:**
+
+```bash
+# Automated Cron Persistence Detection Rule
+# Run every 1 hour, alert on suspicious crontab modifications or commands
+# Optimized: All filters in base search for maximum efficiency
+index=linux sourcetype=auditd earliest=-1h
+file_path IN ("/etc/crontab", "/etc/cron.d/*", "/var/spool/cron/crontabs/*")
+action IN ("modified", "created", "written")
+| join type=left host [
+    search index=linux (sourcetype=linux_secure OR sourcetype=syslog) earliest=-1h "CRON"
+    | rex field=_raw "(?<cron_command>.+)"
+    | eval is_suspicious=if(match(cron_command, "curl|wget|nc|ncat|socat|bash -c|sh -c|base64|/tmp|/dev/shm|python -c|perl -e|/dev/tcp"), "true", "false")
+    | stats values(cron_command) as commands, max(is_suspicious) as suspicious_command by host
+]
+| where (user!="root" AND process_name!="certbot") OR suspicious_command="true"
+| lookup cron_baseline host, user OUTPUT is_baseline
+| where isnull(is_baseline) OR is_baseline="false"
+| eval severity=if(suspicious_command="true", "high", "medium")
+| eval description="Suspicious cron modification detected on ".host." by user ".user
+| table _time, severity, host, user, file_path, action, process_name, commands, description
+```
+
+### Lessons Learned
+
+**What Worked Well:**
+
+- Auditd file integrity monitoring provided comprehensive visibility into crontab modifications
+- Pattern matching on suspicious command strings effectively identified high-risk entries
+- User-based filtering (excluding root) significantly reduced false positives
+- Multi-query approach (file changes + command content + process execution) provided defense-in-depth
+- Manual validation steps confirmed automated findings with 100% accuracy
+
+**What Could Be Improved:**
+
+- Baseline of "known good" cron jobs needed to reduce investigation time
+- Cron command regex extraction unreliable due to inconsistent log formats
+- Better integration between file modification alerts and command content analysis
+- Whitelist management for legitimate tools using suspicious patterns (backup scripts with curl)
+- Query performance optimization - 7-day queries took 15+ seconds on large datasets
+
+**Telemetry Gaps Identified:**
+
+- Dev/staging environments lack auditd monitoring (only 80% production coverage)
+- Bash history logging inconsistently enabled (missing command-line context)
+- No automated cron job inventory for baseline comparison
+- Network connection logging not correlated with cron process execution
+- Missing sysmon for Linux deployment (would provide richer process telemetry)
+
+### Follow-up Actions
+
+- [x] Escalate SOC-2901 to incident response for webadmin user investigation
+- [ ] Create baseline inventory of legitimate cron jobs per host type (target: 2025-11-25)
+- [ ] Deploy auditd to dev/staging environments for complete coverage (INFRA-1457)
+- [ ] Implement automated cron persistence detection rule (DET-0051)
+- [ ] Enable bash history logging across all Linux servers (INFRA-1458)
+- [ ] Create whitelist for known-good cron patterns (backup scripts, monitoring)
+- [ ] Document cron persistence detection playbook for SOC analysts
+- [ ] Schedule recurring hunt execution (monthly)
+
+### Follow-up Hunts
+
+- H-0008: Systemd Timer Persistence Detection (T1053.006)
+- H-0009: Linux Backdoor Analysis on Hosts with Suspicious Crons
+- H-0010: Network Connections from Cron-Spawned Processes
+- H-0011: File Execution from Temporary Directories (/tmp, /dev/shm)
+- H-0012: At Job Persistence Mechanisms (T1053.002)
+
+---
+
+## 📊 Results Showcase
+
+### Detection Timeline
+
+```
+2025-11-17 03:42:15 UTC - Suspicious crontab modification detected
+                         └─> User "webadmin" modified /var/spool/cron/crontabs/webadmin
+                         └─> Auditd file integrity monitoring triggered
+
+2025-11-17 03:42:30 UTC - Automated analysis identifies malicious pattern
+                         └─> Crontab contains curl command to external IP
+                         └─> Schedule: Every 5 minutes (* * * * *)
+                         └─> Target IP: 104.xxx.xxx.23 (not in known-good list)
+
+2025-11-17 03:45:00 UTC - First cron job execution observed
+                         └─> Cron spawned bash process
+                         └─> Downloaded script from external IP
+                         └─> Script attempted to establish reverse shell
+
+2025-11-17 04:12:18 UTC - SOC analyst investigation begins
+                         └─> Incident SOC-2901 created
+                         └─> Crontab confirmed malicious: cryptominer downloader
+
+2025-11-17 04:30:00 UTC - Containment complete
+                         └─> Malicious cron entry removed
+                         └─> webadmin account password reset
+                         └─> Host isolated for forensic analysis
+```
+
+### Query Evolution
+
+**Iteration 1:** File modification-only detection (Too broad)
+- Query: `auditd file modifications to /var/spool/cron/crontabs/*`
+- **Results:** 89 events - many legitimate user cron updates
+- **Problem:** High noise from legitimate automation and user activities
+
+**Iteration 2:** Pattern matching on suspicious commands (Better)
+- Query: Crontab modifications containing curl, wget, bash -c, base64
+- **Results:** 5 events - included backup scripts and monitoring tools
+- **Problem:** False positives from legitimate scripts using similar commands
+
+**Iteration 3:** Behavioral analysis with context (Success!)
+- Query: Suspicious commands + unusual schedules + external network connections
+- **Results:** 1 true positive (cryptominer), 1 false positive (backup script)
+- **Success:** Caught malicious persistence within 30 minutes of creation
+
+### Impact Metrics
+
+| Metric | Value |
+|--------|-------|
+| **Time to Detection** | 30 minutes from crontab modification |
+| **Time to Investigation** | 2.5 hours (manual analysis) |
+| **Time to Containment** | 4.5 hours total |
+| **Hosts Affected** | 1 (web-prod-07) |
+| **Persistence Duration** | ~48 hours (estimated based on C2 logs) |
+| **Cryptominer Impact** | Prevented: Est. $200/month in cloud compute costs |
+| **False Positives** | 1 (backup script using curl for monitoring) |
+| **Coverage Improvement** | Hunt identified 20% auditd gap in dev environments |
+
+### Key Success Factors
+
+1. **Auditd File Integrity Monitoring:** Real-time detection of crontab modifications
+2. **Multi-Context Analysis:** Combined file changes + command patterns + network behavior
+3. **Query Refinement:** 3 iterations reduced FPs from 89 → 1
+4. **Threat Intel Integration:** C2 IP matched known cryptomining campaign (TI-0089)
+5. **Baseline Development:** Created whitelist of 200 known-good cron jobs
+
+### Automated Detection Deployed
+
+Final query converted to real-time detection rule:
+- **Schedule:** Every 5 minutes
+- **Alert Criteria:**
+  - Crontab file modification detected
+  - Command contains network activity keywords (curl, wget, nc)
+  - Schedule is unusually frequent (< 10 minutes)
+  - OR command contains obfuscation (base64, eval, /dev/tcp)
+- **Auto-Response:** Create ticket, snapshot crontab, alert SOC
+- **False Positive Rate:** ~2% (mostly backup scripts, easily whitelisted)
+
+### Lessons Applied from This Hunt
+
+**Detection Improvements:**
+- Created baseline of 200 legitimate cron jobs across environment
+- Developed whitelist for known-good patterns (certbot, backup tools, monitoring)
+- Automated cron inventory collection for ongoing baseline comparison
+
+**Telemetry Enhancements:**
+- Identified 20% coverage gap in dev/staging (auditd not deployed)
+- Added bash history logging to capture command-line context
+- Implemented network connection correlation for cron processes
+
+**Playbook Development:**
+- Documented step-by-step investigation workflow for SOC
+- Created decision tree for cron persistence triage
+- Added automated remediation scripts for common malicious patterns
+
+---
+
+**Hunt Completed:** 2025-11-19
+**Next Review:** 2025-12-19 (recurring monthly hunt)