@lovehlmmm/primekit 1.0.0

package/skills/cti-expert/techniques/username-osint.md

@@ -0,0 +1,227 @@
+# Username OSINT Module
+
+> **Module ID:** USR-OSINT-001
+> **Version:** 1.0.0
+> **Phase:** 5 - Enhancement Modules
+> **Classification:** Cross-Platform Username Enumeration & Profile Correlation
+
+---
+
+## 1. Overview
+
+Username OSINT discovers accounts across platforms using a single handle. Use when a target's username or alias is known and cross-platform presence needs mapping. Combines automated scanning with manual verification to build a digital footprint.
+
+**Key use cases:** Alias tracking, platform correlation, activity timeline reconstruction, alt-account discovery.
+
+---
+
+## 2. Tool Inventory
+
+### Primary — Maigret (3000+ sites)
+```bash
+pip3 install maigret
+# or
+pipx install maigret
+```
+
+### Secondary — Sherlock (400+ sites)
+```bash
+pipx install sherlock-project
+# or
+pip3 install sherlock-project
+```
+
+### Tertiary — Blackbird (600+ sites, async)
+```bash
+pip3 install blackbird-osint
+```
+
+### Web Fallback — WhatsMyName
+- URL: https://whatsmyname.app
+- Repo: https://github.com/WebBreacher/WhatsMyName
+- No install required; browser-based or API-accessible
+
+---
+
+## 3. Investigation Workflow
+
+```
+Step 1: Normalize target username
+  └─ Strip spaces, special chars; note common variants (underscores, dots, numbers)
+
+Step 2: Run Maigret (widest coverage)
+  └─ Use --top-sites 500 for speed vs. --all-sites for coverage
+
+Step 3: Run Sherlock in parallel (different site list)
+  └─ Cross-references sites Maigret may miss
+
+Step 4: Run Blackbird for social-focused sites
+  └─ Better at newer platforms (TikTok, Clubhouse, etc.)
+
+Step 5: Manual spot-check top hits
+  └─ Confirm profile photo, bio, posting style matches target
+
+Step 6: Document and correlate findings
+  └─ Map to real identity, email, location data
+```
+
+---
+
+## 4. CLI Commands & Expected Output
+
+### Maigret
+```bash
+# Top 500 sites, JSON output
+maigret <username> --top-sites 500 --json maigret_results.json
+
+# All sites (slower, ~3000)
+maigret <username> --all-sites --folderoutput ./results/
+
+# With proxy (Tor)
+maigret <username> --top-sites 500 --tor
+```
+
+**Expected output:**
+```
+[+] Checking username "johndoe" on 500 sites
+[+] Twitter: https://twitter.com/johndoe
+[+] GitHub: https://github.com/johndoe
+[-] Facebook: Not found
+[*] Reddit: Claimed (unverified)
+...
+[*] Found 47 accounts. Report saved: johndoe.json
+```
+
+### Sherlock
+```bash
+# Standard scan with JSON output
+sherlock <username> --output sherlock_results.txt --print-found
+
+# Multiple usernames
+sherlock user1 user2 user3 --print-found
+
+# Skip NSFW sites
+sherlock <username> --nsfw
+```
+
+**Expected output:**
+```
+[*] Checking username "johndoe" on: Twitter
+[+] Twitter: https://twitter.com/johndoe
+[+] Instagram: https://www.instagram.com/johndoe/
+[*] GitHub: https://www.github.com/johndoe
+```
+
+### Blackbird
+```bash
+# Basic scan
+blackbird -u <username>
+
+# Export JSON
+blackbird -u <username> --json
+
+# Email-based search
+blackbird -e user@example.com
+```
+
+---
+
+## 5. Fallback Cascade
+
+```
+Maigret unavailable?
+  └─> Use Sherlock (pip3 install sherlock-project)
+
+Sherlock unavailable?
+  └─> Use WhatsMyName web UI at whatsmyname.app
+
+No CLI tools?
+  └─> Manual Google dorking:
+        inurl:<username> site:twitter.com
+        inurl:<username> site:instagram.com
+        "<username>" profile -site:google.com
+
+API rate-limited?
+  └─> Add --timeout 10 --retries 2 flags
+  └─> Use --tor for Maigret to rotate exit nodes
+```
+
+---
+
+## 6. Output Interpretation
+
+### Claim Status Meanings
+```
+[+] Found/Claimed   → Account confirmed active with content
+[*] Available       → Username not taken on that platform
+[-] Not Found       → No account OR profile is private/deleted
+[!] Error           → Timeout, captcha, or platform change
+```
+
+### False Positive Handling
+```
+Common false positives:
+  - 404 pages that still return HTTP 200
+  - Deleted accounts with cached profile URLs
+  - Username squatting (empty accounts)
+
+Verification steps:
+  1. Open URL manually — check for profile photo, bio, posts
+  2. Confirm content matches target's known interests/language
+  3. Check join date vs. known target activity timeline
+  4. Cross-match profile photo via reverse image search (TinEye/Google)
+  5. Look for cross-links (e.g., Instagram bio linking to known Twitter)
+```
+
+### Cross-Platform Correlation
+```
+Strong correlation signals:
+  - Same profile photo across platforms
+  - Same bio text / phrasing
+  - Matching post timestamps (activity overlap)
+  - Cross-posted content or reposts
+  - Same follower/following relationships
+  - Geotags in same locations
+```
+
+---
+
+## 7. Confidence Ratings
+
+| Finding Type | Confidence | Verification Method |
+|---|---|---|
+| Username found on major platform | HIGH | Manual profile review |
+| Username found on niche site | MEDIUM | Cross-check photo/bio |
+| Maigret-only result (no manual check) | LOW | Always verify manually |
+| Cross-platform photo match | HIGH | Reverse image search |
+| Same bio text match | HIGH | Direct comparison |
+| Activity timeline overlap | MEDIUM | Timezone/language check |
+| Deleted/suspended account evidence | MEDIUM | Archive.org, cached pages |
+
+---
+
+## 8. Limitations
+
+1. **Private accounts** — Many platforms hide content; existence detectable but not content
+2. **Username recycling** — Platforms reassign inactive usernames; old hits may be wrong person
+3. **Rate limiting** — Aggressive scanning triggers CAPTCHAs or IP bans; use delays
+4. **Site list drift** — Maigret/Sherlock site databases go stale as platforms change APIs
+5. **Common usernames** — "johndoe", "admin", "user123" produce massive false positive noise
+6. **Regional platforms** — VK, Weibo, Naver poorly covered by Western tools; check manually
+7. **Tor dependency** — --tor flag requires Tor service running locally (`systemctl start tor`)
+
+---
+
+## 9. Command Reference
+
+| Command | Purpose | Input |
+|---|---|---|
+| `/username [handle]` | Full cross-platform username scan | Username string |
+| `/username-variants [handle]` | Generate and scan common variants | Base username |
+| `/username-report [handle]` | Compile correlation report | Username string |
+
+---
+
+*Username OSINT Module v1.0.0*
+*Part of Free OSINT Expert Skill - Phase 5*
+*For authorized investigation and educational purposes only*

package/skills/cti-expert/techniques/web-collection-scrapling.md

@@ -0,0 +1,164 @@
+# Web Collection via Scrapling
+
+> **Module ID:** WEB-COL-001
+> **Version:** 1.0.0
+> **Phase:** Enhancement Module
+> **Classification:** Adaptive Web Data Collection
+
+---
+
+## 1. Overview
+
+Scrapling provides adaptive, resilient web data collection for OSINT. Three fetcher tiers auto-escalate based on site behavior. Headless browser opens BY DEFAULT for JS-heavy targets.
+
+**When to use:** Any time web page content needs to be collected during investigation — replaces manual curl/WebFetch for structured scraping.
+
+---
+
+## 2. Collection Cascade
+
+### Tier 1: Fetcher (Fast Static)
+
+For: Static HTML pages, APIs returning HTML/JSON, simple forms.
+Speed: ~2ms parse time. No browser needed.
+
+```python
+# Install (base, no browser)
+# pip3 install scrapling
+
+from scrapling.fetchers import Fetcher
+
+page = Fetcher.get('https://who.is/whois/example.com')
+registrant = page.css('.whois-data .registrant').text
+
+# Extract all links
+links = [a.attrib['href'] for a in page.css('a[href]')]
+```
+
+### Tier 2: StealthyFetcher (Anti-Bot Bypass)
+
+For: Cloudflare-protected sites, rate-limited services, bot-detection pages.
+Method: Playwright + fingerprint spoofing.
+
+```python
+# Install (requires browser)
+# pip3 install "scrapling[fetchers]" && scrapling install
+
+from scrapling.fetchers import StealthyFetcher
+
+page = StealthyFetcher.get('https://cloudflare-protected-osint.com')
+data = page.css('.results').text
+```
+
+### Tier 3: DynamicFetcher (JavaScript Rendering) — DEFAULT for JS Sites
+
+For: React/Vue/Angular SPAs, infinite scroll, client-rendered content.
+Method: Full Playwright browser with JS execution.
+**THIS IS THE DEFAULT** when target is detected as JS-heavy.
+
+```python
+from scrapling.fetchers import DynamicFetcher
+
+page = DynamicFetcher.get('https://react-osint-tool.com')
+results = page.css('.dynamic-content').getall()
+
+# Wait for specific element before extracting
+page = DynamicFetcher.get('https://spa-target.com', wait_for='div.loaded')
+```
+
+---
+
+## 3. Auto-Escalation Logic
+
+When collecting data from a URL:
+
+1. Try `Fetcher.get(url)` — if response has content, use it
+2. If 403/429/captcha detected → escalate to `StealthyFetcher`
+3. If content empty or JS-placeholder detected → escalate to `DynamicFetcher`
+4. If all fail → fall back to WebFetch/WebSearch
+5. Tag finding: `[scrapling-static]` / `[scrapling-stealth]` / `[scrapling-dynamic]`
+
+**JS-heavy detection heuristic:**
+- Response body contains `<div id="root"></div>` or `<div id="app"></div>` with no content
+- Response body < 1KB but Content-Length header suggests larger page
+- Known JS-heavy domains: social media, modern OSINT tools, dashboards
+
+---
+
+## 4. Headless Browser Auto-Open Policy
+
+DEFAULT BEHAVIOR when a URL is fetched during investigation:
+
+| Condition | Fetcher Used | Browser? |
+|-----------|-------------|----------|
+| Static content (HTML with data) | Fetcher | No |
+| 403/bot-block response | StealthyFetcher | Headless (stealth) |
+| JS-heavy/SPA detected | DynamicFetcher | Headless (full render) |
+| Screenshot needed | agent-browser (Playwright) | Full browser |
+
+---
+
+## 5. Session Management
+
+For multi-page collection (paginated results, login-required):
+
+```python
+from scrapling.fetchers import StealthyFetcher
+
+# Session persists cookies across requests
+session = StealthyFetcher()
+page1 = session.get('https://target.com/results?page=1')
+page2 = session.get('https://target.com/results?page=2')
+# Cookies, headers persist automatically
+```
+
+---
+
+## 6. Adaptive Selectors
+
+Scrapling's adaptive selectors auto-relocate when site structure changes. Critical for OSINT durability — selectors that worked last week keep working even if the target site redesigns.
+
+```python
+# First run: learns selector location
+page = Fetcher.get('https://target.com')
+name = page.css('#profile-name').text  # Scrapling memorizes context
+
+# Later: site changes #profile-name to .user-name
+# Scrapling auto-finds the equivalent element
+```
+
+---
+
+## 7. Integration with Existing Techniques
+
+Scrapling enhances these investigation modules:
+
+| Technique | Enhancement |
+|-----------|-------------|
+| `scam-check.md` | Steps 2-7: PhishTank, CheckPhish scraping |
+| `phone-osint.md` | USPhoneBook scraping (replaces cloudscraper) |
+| `fx-visitor-intelligence.md` | SimilarWeb data extraction |
+| `image-forensics` | FaceCheck.id result scraping |
+| `social-media-platforms.md` | Profile data extraction |
+| `whois-universal.md` | who.is web scrape fallback (Layer 4) |
+
+---
+
+## 8. Full Fallback Cascade
+
+```
+agent-browser (Playwright) → DynamicFetcher → StealthyFetcher → Fetcher → WebFetch → WebSearch → curl
+```
+
+---
+
+## 9. Confidence Ratings
+
+| Collection Method | Tag | Confidence |
+|-------------------|-----|-----------|
+| Scrapling Fetcher | [scrapling-static] | HIGH |
+| Scrapling StealthyFetcher | [scrapling-stealth] | HIGH |
+| Scrapling DynamicFetcher | [scrapling-dynamic] | HIGH |
+| WebFetch | [fetch] | MEDIUM |
+| WebSearch | [search] | MEDIUM |
+| curl | [manual] | MEDIUM |

package/skills/cti-expert/techniques/web-dns-forensics.md

@@ -0,0 +1,293 @@
+# Web & DNS Forensics Module
+
+> **Module ID:** WEB-DNS-001
+> **Version:** 1.0.0
+> **Phase:** 5 - Enhancement Modules
+> **Classification:** Web Platform & DNS Investigation Techniques
+
+---
+
+## 1. Overview
+
+Techniques for extracting intelligence from web platforms, DNS records, and online services. Covers Google Docs/Sheets enumeration, DNS zone transfers, Tor relay lookups, GitHub repository analysis, Telegram bot investigation, FEC research, WHOIS deep investigation, and Google Image advanced filters.
+
+---
+
+## 2. Google Image TBS Filters
+
+Append `&tbs=` parameters to Google Image search URLs for precision filtering:
+
+| Filter | Parameter | Use Case |
+|--------|-----------|----------|
+| Faces only | `itp:face` | Profile photos — strips logos/banners |
+| Clipart | `itp:clipart` | Logos, icons |
+| Animated GIF | `itp:animated` | Animated images |
+| Specific color | `ic:specific,isc:green` | Dominant color filter |
+| Transparent BG | `ic:trans` | PNGs with transparency |
+| Large images | `isz:l` | High resolution only |
+| Min resolution | `isz:lt,islt:2mp` | Greater than 2 megapixels |
+
+**Combined example — LinkedIn face photos:**
+```
+https://www.google.com/search?q="company"+"intern"+site:linkedin.com&tbm=isch&tbs=itp:face
+```
+
+The `itp:face` filter is especially useful — combine with `site:` and `after:YYYY-MM-DD` for targeted recon.
+
+---
+
+## 3. Google Docs/Sheets Enumeration
+
+When targets share Google Docs/Sheets links, try these access URLs:
+
+| URL Suffix | Purpose |
+|-----------|---------|
+| `/export?format=csv` | Export as CSV |
+| `/pub` | Published version |
+| `/gviz/tq?tqx=out:csv` | Visualization API CSV export |
+| `/htmlview` | HTML view |
+
+Sheet IDs are stable identifiers even if sharing settings change. Private sheets require authentication.
+
+### 3.1 Xeuledoc — Google Document Metadata Extraction
+
+**Tool:** [xeuledoc](https://github.com/Malfrats/xeuledoc) (Python, GPLv3)
+
+Extracts metadata and permissions from any publicly shared Google document without authentication. Queries Google Drive's internal API using the document ID.
+
+**Supported document types:** Docs, Sheets, Slides, Drawings, Drive files, My Maps, Apps Script, Jamboard.
+
+**Installation:**
+```bash
+pip3 install xeuledoc
+```
+
+**Usage:**
+```bash
+# Analyze any Google document share link
+xeuledoc "https://docs.google.com/spreadsheets/d/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms"
+```
+
+**Extracted data:**
+
+| Field | Forensic Value | Confidence |
+|-------|---------------|------------|
+| Document ID | Persistent identifier (33 or 44 chars) | HIGH |
+| Creation date | Document origin timestamp (UTC) | HIGH |
+| Last edit date | Most recent modification (UTC) | HIGH |
+| Public permissions | Viewer/editor/commenter roles | HIGH |
+| Owner name | Document creator identity | HIGH |
+| Owner email | Creator email address | HIGH |
+| Owner Google ID | Persistent Google account identifier | HIGH |
+
+**OSINT value:**
+- Reveals document owner identity (name, email, Google ID) even when the doc itself shows no author
+- Creation vs edit date gap exposes document lifecycle and editing patterns
+- Public permissions reveal whether the target intentionally shared or misconfigured access
+- Owner Google ID is a persistent identifier useful for cross-referencing across Google services
+
+**Integration with case model:**
+- Owner email → register as EMAIL subject → feed to `/email-deep` and `/breach-deep`
+- Owner name → register as PERSON subject → feed to `/username` enumeration
+- Document → register as ASSET subject with `owns` connection to owner
+- Timestamps → feed to `/timeline`
+
+**Fallback cascade:**
+1. xeuledoc (primary — automated, structured output)
+2. Manual Google Drive API query with document ID (secondary)
+3. Google cache / Wayback Machine snapshot of the document (tertiary)
+
+**Limitations:**
+- Only works on public or anyone-with-link documents; private docs return error
+- Rate-limited by Google — automatic retries (up to 100) with progress display
+- Metadata can theoretically be manipulated; treat as findings, not ground truth
+
+---
+
+## 4. DNS Zone Transfers & TXT Record OSINT
+
+### Zone Transfer Attempt
+
+```bash
+dig axfr @ns.domain.com domain.com
+```
+
+### TXT Record Mining
+
+```bash
+dig -t txt subdomain.ctf.domain.com
+dig -t txt _dmarc.domain.org
+dig -t any domain.org
+```
+
+**Key insight:** DNS TXT records are publicly queryable. Always check TXT, CNAME, MX for target domains and subdomains. Flags/data often in TXT records of subdomains, not root domain.
+
+---
+
+## 5. Tor Relay Lookups
+
+Identify Tor relay information from fingerprints:
+
+```
+https://metrics.torproject.org/rs.html#simple/<FINGERPRINT>
+```
+
+- Check family members (relays operated by same entity)
+- Sort by "first seen" date for chronological ordering
+
+**String identification:** 40 hex chars = SHA-1 (Tor fingerprint), 64 hex = SHA-256, 32 hex = MD5.
+
+---
+
+## 6. GitHub Repository Analysis
+
+Hidden information often exists in GitHub repo comments, not just code:
+
+```bash
+# Issue comments
+gh api repos/OWNER/REPO/issues/comments
+
+# All commits
+gh api repos/OWNER/REPO/commits
+
+# PR review comments
+gh api repos/OWNER/REPO/pulls/comments
+
+# Wiki pages (if enabled)
+# Clone wiki: git clone https://github.com/OWNER/REPO.wiki.git
+```
+
+Check: issue comments, PR reviews, commit messages, wiki edit history.
+
+---
+
+## 7. Telegram Bot Investigation
+
+### Finding Bot References
+
+```python
+# Search browser history for Telegram URLs
+import sqlite3
+conn = sqlite3.connect("History")  # Edge/Chrome history DB
+cur = conn.cursor()
+cur.execute("SELECT url FROM urls WHERE url LIKE '%t.me/%'")
+```
+
+### Bot Interaction Workflow
+
+1. Visit `https://t.me/<botname>` → Opens in Telegram
+2. Start conversation with `/start`
+3. Bot may require verification (CTF-style challenges)
+4. Answers often require knowledge from forensic analysis
+
+### Verification Question Patterns
+
+- "Which user account did you use for X?" → Check browser history, login records
+- "Which account was modified?" → Check Security.evtx Event 4781 (rename)
+- "What file did you access?" → Check MRU, Recent files, Shellbags
+
+Bot responses may reveal: attacker identity, credentials to secondary systems, direct flag components, links to hidden web services.
+
+---
+
+## 8. FEC Political Donation Research
+
+Track organizational donors through FEC filings.
+
+**Resources:**
+- [FEC.gov](https://www.fec.gov/data/) — Committee receipts and expenditures
+- 501(c)(4) organizations can donate to Super PACs without disclosing original funders
+
+**Technique:** Look for largest organizational donors, then research org leadership (CEO/President).
+
+---
+
+## 9. WHOIS Deep Investigation
+
+> **Moved to dedicated module:** See `techniques/whois-universal.md` for the full
+> multi-TLD WHOIS cascade with support for all ccTLDs (.vn, .th, .sg, .kr, etc.),
+> reverse WHOIS, historical WHOIS, and IP/ASN lookups — all free, no API keys.
+>
+> Covers: 4-layer fallback (whoisdomain → CLI whois → Whoxy API → web scrape),
+> 27+ ccTLD server mappings, .vn deep dive, confidence ratings.
+
+---
+
+## 10. Wayback Machine CDX API
+
+```bash
+# Find all archived URLs for a site
+curl "http://web.archive.org/cdx/search/cdx?url=example.com*&output=json&fl=timestamp,original,statuscode"
+
+# Check Twitter profile archives
+curl "http://web.archive.org/cdx/search/cdx?url=twitter.com/USERNAME*&output=json"
+
+# Check t.co shortlinks
+curl "http://web.archive.org/cdx/search/cdx?url=t.co/SHORTCODE&output=json"
+```
+
+---
+
+## 11. Unicode Homoglyph Steganography
+
+Visually-identical Unicode characters from different blocks encode binary data in text.
+
+### Detection
+
+- Text looks normal but character-by-character analysis reveals non-ASCII codepoints
+- Characters from Cyrillic, Greek, Armenian, Mathematical Monospace blocks
+- Each character encodes 1 bit: ASCII = 0, homoglyph = 1
+
+### Common Homoglyph Pairs
+
+| ASCII | Homoglyph | Unicode Block |
+|-------|-----------|---------------|
+| `a` (U+0061) | `а` (U+0430) | Cyrillic |
+| `o` (U+006F) | `о` (U+043E) | Cyrillic |
+| `e` (U+0065) | `е` (U+0435) | Cyrillic |
+| `s` (U+0073) | `ѕ` (U+0455) | Cyrillic DZE |
+| `p` (U+0070) | `р` (U+0440) | Cyrillic |
+
+### Decoding
+
+```python
+def decode_homoglyph_stego(text):
+    bits = []
+    for ch in text:
+        if ch in ('\u2019',):  # Platform auto-inserted right single quote
+            continue
+        if ord(ch) < 128:
+            bits.append(0)  # Standard ASCII
+        else:
+            bits.append(1)  # Unicode homoglyph = 1 bit
+
+    flag = ''
+    for i in range(0, len(bits) - 7, 8):
+        byte_val = 0
+        for j in range(8):
+            byte_val = (byte_val << 1) | bits[i + j]
+        flag += chr(byte_val)
+    return flag
+```
+
+**Key lessons:** Platform auto-formatting (smart quotes) must be excluded from bit encoding. Check ALL replies/posts, not just main content.
+
+---
+
+## 12. Confidence Ratings
+
+| Finding | Confidence | Notes |
+|---------|-----------|-------|
+| DNS TXT record data | HIGH | Authoritative DNS response |
+| WHOIS registration data | HIGH | Registrar-verified |
+| Wayback Machine archive | HIGH | Timestamped snapshot |
+| GitHub commit/issue data | HIGH | Platform audit trail |
+| Tor relay fingerprint match | HIGH | Cryptographic identity |
+| FEC filing data | HIGH | Government records |
+| Homoglyph steganography decode | HIGH | Mathematical decoding |
+| Reverse WHOIS correlation | MEDIUM | May be shared hosting |
+
+---
+
+*Web & DNS Forensics Module v1.0.0*
+*Part of Free OSINT Expert Skill - Phase 5*