@lovehlmmm/primekit 1.0.0

package/skills/cti-expert/analysis/archive-explorer.md

@@ -0,0 +1,370 @@
+# Archive Explorer
+
+Techniques for recovering subject history from web archives. Primary tool: **waymore** (multi-source archive mining). Secondary: Common Crawl API, Internet Archive CDX, Archive.today.
+
+---
+
+## Source Priority
+
+| Source | Coverage | Reliability | Best For |
+|---|---|---|---|
+| Common Crawl | Broad web, frequent crawls | Medium-High | Domain content, public pages |
+| Internet Archive Wayback | Deep historical, social profiles | High | Timeline reconstruction |
+| Archive.today | Snapshot-on-demand, JS-rendered | High | Current snapshots, social posts |
+| Google cache | Last crawl only | Low | Recent content, quick check |
+| Cached APIs (Twitter, Reddit) | Platform-specific | Medium | Social content before deletion |
+
+---
+
+## 0. Waymore — Primary Archive Mining Tool
+
+**waymore** (https://github.com/xnl-h4ck3r/waymore) queries 7 archive sources simultaneously and optionally downloads archived response bodies for deep content analysis. Replaces manual querying of individual archive APIs.
+
+**Install:** `pip3 install waymore`
+
+**Sources queried (all 7):**
+1. Wayback Machine (web.archive.org)
+2. Common Crawl (106+ index collections)
+3. AlienVault OTX (otx.alienvault.com)
+4. URLScan (urlscan.io — requires free API key)
+5. VirusTotal (virustotal.com)
+6. GhostArchive (ghostarchive.org)
+7. Intelligence X (intelx.io — paid tiers)
+
+**Key CLI patterns:**
+```bash
+# Get all archived URLs for a domain (including subdomains):
+waymore -i <domain> -mode U -oU waymore-urls.txt
+
+# Get URLs only for the base domain (no subs):
+waymore -i <domain> -mode U -n -oU waymore-urls.txt
+
+# Download archived response bodies (for content mining):
+waymore -i <domain> -mode R -oR ./waymore-responses/
+
+# Both URLs and responses:
+waymore -i <domain> -mode B -oU waymore-urls.txt -oR ./waymore-responses/
+
+# Filter by date range:
+waymore -i <domain> -mode U -from 2020 -to 2024 -oU waymore-urls.txt
+
+# Filter by MIME type (e.g., only HTML):
+waymore -i <domain> -mode U -mt text/html -oU waymore-urls.txt
+
+# Keywords-only mode (faster — regex filter):
+waymore -i <domain> -mode U -ko "api|admin|login|password" -oU waymore-urls.txt
+
+# Limit Common Crawl collections queried (speed vs coverage):
+waymore -i <domain> -mode U -lcc 5 -oU waymore-urls.txt
+```
+
+**Why waymore over GAU:**
+- Queries 7 sources vs GAU's 4 (Wayback, OTX, Common Crawl, URLScan)
+- Downloads actual response bodies, not just URL listings
+- Handles rate limiting with retries and backoff
+- Outputs deduplicated, filterable results
+- Supports date-bounded and MIME-filtered queries
+
+**Fallback:** If waymore unavailable, use GAU + manual Wayback CDX queries as documented below.
+
+---
+
+## 1. Common Crawl Access
+
+Common Crawl provides petabyte-scale raw web data and an index API. Prefer over Wayback for breadth; use Wayback for depth on specific URLs.
+
+**CDX Index API (Common Crawl):**
+```
+https://index.commoncrawl.org/CC-MAIN-{YYYY-MM}/cdx?url={domain}&output=json
+```
+
+Available crawl indexes: quarterly releases, labeled `CC-MAIN-YYYY-WW`.
+
+```python
+def query_commoncrawl(url, crawl_id="CC-MAIN-2024-10"):
+    endpoint = f"https://index.commoncrawl.org/{crawl_id}/cdx"
+    params = {
+        "url": url,
+        "output": "json",
+        "fields": "timestamp,url,mime,status,filename,offset,length"
+    }
+    response = requests.get(endpoint, params=params)
+    return [json.loads(line) for line in response.text.strip().split('\n')]
+
+# Fetch the actual WARC content:
+def fetch_warc_record(filename, offset, length):
+    warc_url = f"https://data.commoncrawl.org/{filename}"
+    headers = {"Range": f"bytes={offset}-{offset+length-1}"}
+    return requests.get(warc_url, headers=headers)
+```
+
+---
+
+## 2. Internet Archive CDX API
+
+The CDX API returns structured capture metadata. More granular than the Wayback UI.
+
+**All captures for a URL:**
+```
+https://web.archive.org/cdx/search/cdx?url={url}&output=json&limit=500
+```
+
+**Date-bounded capture list:**
+```
+https://web.archive.org/cdx/search/cdx?url={url}&from={YYYYMMDD}&to={YYYYMMDD}&output=json
+```
+
+**Filter by HTTP status:**
+```
+https://web.archive.org/cdx/search/cdx?url={url}&filter=statuscode:200&output=json
+```
+
+**Snapshot URL formats:**
+
+| Format | URL Pattern | Purpose |
+|---|---|---|
+| Standard | `web/{ts}/{url}` | As captured with Wayback toolbar |
+| Clean | `web/{ts}if_/{url}` | No injected Wayback JS |
+| Text extract | `web/{ts}id_/{url}` | Raw text, no assets |
+
+---
+
+## 3. Social Platform Archive Patterns
+
+| Platform | Wayback Pattern |
+|---|---|
+| Twitter / X | `web/*/https://twitter.com/{handle}` or `/status/{id}` |
+| Instagram | `web/*/https://www.instagram.com/{handle}/` or `/p/{shortcode}/` |
+| Reddit | `web/*/https://www.reddit.com/user/{username}/` |
+| LinkedIn | `web/*/https://www.linkedin.com/in/{slug}/` |
+| Facebook | `web/*/https://www.facebook.com/{username}` |
+
+---
+
+## 4. Timeline Reconstruction
+
+Build a chronological event log from archive captures.
+
+**Phase 1 — Capture inventory:**
+```python
+def collect_captures(url):
+    captures = query_wayback_cdx(url)
+    captures += query_commoncrawl(url)
+    return deduplicate_by_timestamp(captures)
+```
+
+**Phase 2 — Event classification:**
+```python
+EVENT_TYPES = {
+    "first_seen":        lambda c: c.is_earliest,
+    "profile_update":    lambda c: profile_fields_changed(c, prev),
+    "content_added":     lambda c: new_posts_detected(c, prev),
+    "content_removed":   lambda c: posts_missing_vs_prev(c, prev),
+    "username_change":   lambda c: c.handle != prev.handle,
+    "archive_gap":       lambda c: days_since_prev(c, prev) > 30,
+    "account_suspended": lambda c: c.http_status in (404, 302) and prev.status == 200,
+}
+```
+
+**Phase 3 — Timeline format:**
+```
+{YYYY-MM-DD} [EVENT_TYPE] description — confidence X%
+```
+
+Example:
+```
+2019-02-14 [FIRST_SEEN]     Account created (earliest capture)
+2019-08-03 [CONTENT_ADDED]  Campaign posts begin (12 posts in 1 week)
+2020-04-17 [PROFILE_UPDATE] Username: handle_A → handle_B
+2020-11-22 [CONTENT_REMOVED] 8 posts deleted — archived copies available
+2021-06-30 [ARCHIVE_GAP]    47-day gap — possible suspension
+2021-08-16 [FIRST_SEEN]     Account re-appears under handle_B
+```
+
+---
+
+## 5. Domain Archive Exploration
+
+**Registration history:**
+```
+site:web.archive.org {domain} whois
+site:domaintools.com/research/whois-history/search/ (manual)
+```
+
+**DNS record history:**
+```
+site:securitytrails.com/domain/{domain}/history/a
+site:viewdns.info/iphistory/?domain={domain}
+```
+
+**Subdomain discovery via archives:**
+```python
+def find_subdomains_in_archive(domain):
+    cdx_url = f"https://web.archive.org/cdx/search/cdx?url=*.{domain}&output=json&fl=original"
+    results = requests.get(cdx_url).json()
+    return set(extract_subdomain(r[0]) for r in results[1:])
+```
+
+---
+
+## 6. Authenticity Verification
+
+Before citing archive content as a finding:
+
+| Check | Method |
+|---|---|
+| Timestamp integrity | Verify timestamp in URL matches page date references |
+| Wayback injection | Look for Wayback toolbar HTML; use `if_` variant to strip |
+| Cross-archive consistency | Same content in Archive.today and Common Crawl |
+| Image hash validation | Compare archived images to other sources |
+
+Source confidence: Wayback 88%, Archive.today 83%, Common Crawl 76%, Google cache 58% (ephemeral).
+
+---
+
+## 7. Recovery Dorks
+
+```
+# Find archived copies of a specific URL
+site:web.archive.org "{target_url}"
+site:archive.today "{target_url}"
+
+# Find archived content by keyword
+site:web.archive.org "{exact_phrase}" "{domain}"
+
+# Recover deleted page (exists in archive, 404 now)
+site:web.archive.org/web/*/https://{domain}/{path}
+
+# Find historical mentions of username
+site:web.archive.org "{handle}" site:{platform}.com
+```
+
+---
+
+## Cross-References
+
+- `analysis/drift-monitor.md` — triggers archive recovery on DRIFT_CRITICAL
+- `techniques/deletion-recovery.md` — multi-source recovery workflow
+- `engine/archive-cache.md` — local snapshot storage
+
+---
+
+## 8. Snapshot Listing
+
+```
+/snapshots [url] [--from YYYY] [--to YYYY]
+```
+
+Lists all available Wayback Machine snapshots for a URL with timestamps and HTTP status codes. Useful for identifying capture density, finding the earliest/latest record, and selecting specific versions for diff comparison.
+
+**Options:**
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `[url]` | required | Full URL or domain to query |
+| `--from YYYY` | earliest available | Start year filter |
+| `--to YYYY` | current year | End year filter |
+
+**Implementation — Wayback CDX API:**
+
+```bash
+# Via waymore (preferred — multi-source):
+waymore -i <url> -mode U -from <YYYY> -to <YYYY> -oU snapshots.txt
+
+# Direct CDX fallback:
+curl "https://web.archive.org/cdx/search/cdx?url=<url>&from=<YYYYMMDD>&to=<YYYYMMDD>&output=json&fl=timestamp,statuscode,mimetype&limit=200"
+```
+
+**Output format:**
+
+```
+SNAPSHOTS — {url}
+Range: {from_year} → {to_year}  |  Total captures: {n}
+════════════════════════════════════════════════════════════
+  Timestamp (UTC)      Status  MIME
+  ──────────────────   ──────  ──────────────────
+  2019-02-14 08:03     200     text/html
+  2019-08-22 14:17     200     text/html
+  2020-04-11 03:55     301     —
+  2021-06-30 09:40     404     —
+  2023-11-05 16:22     200     text/html
+  ...
+════════════════════════════════════════════════════════════
+First capture : {earliest_timestamp}
+Last capture  : {latest_timestamp}
+Status 200    : {n_200}  |  3xx: {n_3xx}  |  4xx/5xx: {n_err}
+```
+
+Status codes are color-coded in terminal: 200 green, 3xx yellow, 4xx/5xx red.
+
+---
+
+## 9. Archive Content Diff
+
+```
+/diff [url] [--from <date>] [--to <date>]
+```
+
+Fetches two archived versions of a URL and computes a text diff, highlighting significant content changes. Used to detect when sensitive information was added or removed, page structure shifts, or content manipulation.
+
+**Options:**
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `[url]` | required | URL to compare |
+| `--from <date>` | earliest snapshot | Date of the "before" version (YYYY-MM-DD or YYYYMMDDHHMMSS) |
+| `--to <date>` | latest snapshot | Date of the "after" version |
+
+If exact timestamps are not provided, the nearest available snapshot to each date is used.
+
+**Implementation:**
+
+```python
+def fetch_clean_snapshot(url, timestamp):
+    # Use if_ variant to strip Wayback toolbar injection
+    wayback_url = f"https://web.archive.org/web/{timestamp}if_/{url}"
+    resp = requests.get(wayback_url, timeout=15)
+    soup = BeautifulSoup(resp.text, "html.parser")
+    # Remove script/style noise
+    for tag in soup(["script", "style", "nav", "footer"]):
+        tag.decompose()
+    return soup.get_text(separator="\n", strip=True)
+
+def compute_diff(text_before, text_after):
+    import difflib
+    lines_a = text_before.splitlines(keepends=True)
+    lines_b = text_after.splitlines(keepends=True)
+    return list(difflib.unified_diff(lines_a, lines_b, lineterm=""))
+```
+
+**ASCII diff report:**
+
+```
+ARCHIVE DIFF — {url}
+Before : {from_timestamp} (snapshot {wayback_ts_a})
+After  : {to_timestamp}   (snapshot {wayback_ts_b})
+════════════════════════════════════════════════════════════
+Changes: +{lines_added} added  -{lines_removed} removed  ~{lines_changed} modified
+
+SIGNIFICANT CHANGES (flagged by keyword heuristic):
+  [+] Line 42  : "Contact: admin@newdomain.com"          ← email added
+  [-] Line 87  : "Hosted by: LegitProvider Inc"          ← provider removed
+  [+] Line 103 : "Terms updated: 2023-11-01"             ← date change
+
+FULL DIFF (unified format):
+--- before  {wayback_ts_a}
++++ after   {wayback_ts_b}
+@@ -{from_line},{ctx} +{to_line},{ctx} @@
+ unchanged context line
+-removed line
++added line
+ unchanged context line
+════════════════════════════════════════════════════════════
+```
+
+**Significance heuristic** — flags lines containing: email patterns, IP addresses, domain names, dates, phone numbers, monetary values, credential keywords (`password`, `token`, `key`, `secret`).
+
+**Cross-references:**
+- `/snapshots` — use first to identify candidate timestamps before running `/diff`
+- `analysis/drift-monitor.md` — automated diff triggers on DRIFT_CRITICAL
+- `engine/finding-framework.md` — significant diff changes create findings with type `infrastructure`

package/skills/cti-expert/analysis/auto-branch-rules.md

@@ -0,0 +1,198 @@
+# Auto-Branch Rules
+
+Rules governing when the workspace should automatically expand to a new subject or connection.
+
+---
+
+## Rule Format
+
+All rules follow: WHEN [condition] THEN [action] BECAUSE [rationale]
+
+---
+
+## 1. Identity Branch Rules
+
+**IBR-01: Leet-Encoded Handle**
+```
+WHEN subject.handle CONTAINS leet_substitution (0,1,3,4,5,7,@,$)
+AND normalized_handle EXISTS on >=1 additional platform
+THEN branch(type=HANDLE_VARIANT, target=normalized_handle, priority=HIGH)
+BECAUSE leet encoding is a low-effort evasion tactic indicating
+         the subject is aware of cross-platform linking
+confidence = 83%
+```
+
+**IBR-02: Sequential Handle Suffix**
+```
+WHEN subject.handle MATCHES regex [a-z]+\d{1,3}$
+AND handle_minus_suffix EXISTS on same or other platform
+THEN branch(type=HANDLE_SERIES, target=base_handle, priority=MEDIUM)
+BECAUSE numbered suffixes indicate account iteration after
+         suspension, ban, or deactivation
+confidence = 60%
+```
+
+**IBR-03: Platform Gap**
+```
+WHEN subject ACTIVE on platform_A
+AND similar_handle INACTIVE on platform_A
+AND similar_handle ACTIVE on platform_B
+THEN branch(type=PLATFORM_MIGRATION, target=platform_B_handle, priority=MEDIUM)
+BECAUSE subjects often migrate while preserving identity signals
+confidence = 68%
+```
+
+---
+
+## 2. Credential Branch Rules
+
+**CBR-01: Display Name / Email Mismatch**
+```
+WHEN subject.display_name = "Name_A"
+AND subject.email CONTAINS name_token NOT IN known_aliases
+THEN branch(type=ALIAS_INVESTIGATION, target=name_token, priority=HIGH)
+BECAUSE email name token predates current display name and may
+         reflect the subject's primary identity
+confidence = 74%
+```
+
+**CBR-02: Disposable Email on Verified Platform**
+```
+WHEN subject.email_domain IN disposable_provider_list
+AND platform.verification_tier = "professional"
+THEN branch(type=EMAIL_PREFIX_TRACE, target=email_prefix, priority=HIGH)
+BECAUSE using throwaway email on a trust-requiring platform
+         indicates deliberate identity compartmentalization
+confidence = 79%
+```
+
+**CBR-03: Domain Affiliation Mismatch**
+```
+WHEN subject.email_domain != subject.stated_employer_domain
+AND subject.email_domain NOT IN personal_provider_list
+THEN branch(type=DOMAIN_INVESTIGATION, target=email_domain, priority=MEDIUM)
+BECAUSE unusual domain affiliation may indicate undisclosed
+         organizational ties or role
+confidence = 67%
+```
+
+---
+
+## 3. Social Graph Branch Rules
+
+**SBR-01: High Co-mention Frequency**
+```
+WHEN subject_A mentions subject_B >= 6 times
+AND subject_B mentions subject_A >= 3 times
+AND window = last_60_days
+THEN branch(type=CO_SUBJECT_EXPANSION, target=subject_B, priority=MEDIUM)
+BECAUSE bidirectional mention volume above threshold signals
+         an active operational or personal connection
+confidence = 73%
+```
+
+**SBR-02: Photo Tag Cluster**
+```
+WHEN subject_A tagged_with subject_B in images >= 3 times
+THEN branch(type=PHYSICAL_CO_PRESENCE, target=subject_B, priority=HIGH)
+BECAUSE repeated photo co-tagging implies physical proximity
+         and direct relationship
+confidence = 84%
+```
+
+**SBR-03: Coordinated Follower Overlap**
+```
+WHEN follower_set_A ∩ follower_set_B >= 50 accounts
+AND quality_score(intersection) < 35
+THEN branch(type=COORDINATION_INVESTIGATION, target="shared_follower_cluster", priority=HIGH)
+BECAUSE low-quality shared followers are a signature of
+         coordinated inauthentic behavior
+confidence = 78%
+```
+
+**SBR-04: Shared Niche Community Membership**
+```
+WHEN subject_A.communities ∩ subject_B.communities >= 3
+AND all_communities.specificity = "niche"
+THEN branch(type=COMMUNITY_AFFILIATION, target=subject_B, priority=LOW)
+BECAUSE niche community overlap provides weak but accumulative
+         finding of shared interests or associations
+confidence = 58%
+```
+
+---
+
+## 4. Geographic Branch Rules
+
+**GBR-01: Location Impossibility**
+```
+WHEN two_posts FROM same_subject
+AND location_distance > 500km
+AND time_between < 90_minutes
+THEN branch(type=GEO_ANOMALY_INVESTIGATION, target="VPN/proxy or account sharing", priority=HIGH)
+BECAUSE physics rules out legitimate travel; account may be
+         shared or proxied
+confidence = 89%
+```
+
+---
+
+## 5. Branch Priority Matrix
+
+| Branch Type | Confidence Threshold | Priority | Max Expansion Per Session |
+|---|---|---|---|
+| Handle exact match (other platform) | ≥95% | CRITICAL | Unlimited |
+| Email exact match | ≥95% | CRITICAL | Unlimited |
+| Profile image match | ≥90% | HIGH | 5 |
+| Handle variant (leet/series) | ≥83% | HIGH | 5 |
+| Geographic impossibility | ≥89% | HIGH | 3 |
+| Coordinated follower overlap | ≥78% | HIGH | 3 |
+| Co-mention bidirectional | ≥73% | MEDIUM | 5 |
+| Domain mismatch | ≥67% | MEDIUM | 3 |
+| Niche community overlap | ≥58% | LOW | 2 |
+
+---
+
+## 6. Branch Suppression
+
+```
+WHEN confidence < 50%
+AND corroborating_findings < 2
+THEN suppress_branch
+LOG = "suppressed: below minimum finding threshold"
+
+WHEN branch_target IN workspace.already_expanded
+THEN suppress_branch
+LOG = "suppressed: target already in scope"
+
+WHEN branch_target.privacy_tier = "protected"
+AND case.authorization != "confirmed"
+THEN suppress_branch
+LOG = "suppressed: authorization not confirmed for target"
+```
+
+---
+
+## 7. Dorks for Branch Validation
+
+```
+# Validate handle variant
+"[base_handle]" OR "[leet_variant]" site:twitter.com OR site:instagram.com
+
+# Validate email prefix as alias
+"[email_prefix]" -site:[original_platform]
+
+# Validate photo co-presence
+site:instagram.com "[subject_A]" "[subject_B]"
+
+# Validate coordinated following
+site:twitter.com/[subjectA]/following  (compare against subjectB manually)
+```
+
+---
+
+## Cross-References
+
+- `analysis/cross-reference-engine.md` — scoring connections between branched subjects
+- `analysis/deviation-detector.md` — IDENTITY_DRIFT triggers that may initiate branches
+- `techniques/username-enumeration.md` — handle variant generation