voidaccess 1.4.7__tar.gz → 1.6.0__tar.gz

{voidaccess-1.4.7/voidaccess.egg-info → voidaccess-1.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: voidaccess
-Version: 1.4.7
+Version: 1.6.0
 Summary: Dark web OSINT CLI — automated threat intelligence from query to report
 Author: VoidAccess
 License-Expression: MIT
@@ -65,6 +65,35 @@ Commercial threat intelligence platforms often charge prohibitive annual fees fo
 
 ---
 
+## What's New in v1.6.0
+
+- **Optional clearnet scraping proxy (ScrapingAnt)** — paste sites and RSS feeds can now be routed through ScrapingAnt. Affects clearnet scraping only; Tor, `.onion`, GitHub, and GitLab traffic are never affected.
+- **Two mutually exclusive transports** — pick one, not both:
+  - **REST API transport** — `VOIDACCESS_USE_PROXIES=true` (legacy v1.5.0 toggle) routes requests through ScrapingAnt's Web Scraping API.
+  - **Proxy Mode transport** — `VOIDACCESS_USE_PROXY=true` routes requests through ScrapingAnt's HTTP CONNECT endpoint at `proxy.scrapingant.com:8080`.
+  - Per [ScrapingAnt docs](https://docs.scrapingant.com/proxy-mode): "Proxy Mode is a light front-end for the scraping API and has all the same functionality and performance" — so the two are alternate transports to the same backend, never chained.
+- **`SCRAPINGANT_PROXY_TYPE`** — `residential` (default) or `datacenter`; per docs this is passed as a `proxy_type=` parameter in the Proxy Mode username string (which is built at connection time as `"scrapingant&browser=false&proxy_type=..."`). NOT a separate hostname.
+- **Single credential** — `SCRAPINGANT_API_KEY` is the only real credential; the Proxy Mode username is a literal constant per docs. No per-customer username field, no second key.
+- **New CLI surfaces** — `voidaccess configure proxy` now prompts for key + type in one uninterrupted block, plus `--enable-proxy / --disable-proxy` for non-interactive Proxy Mode toggling and `--show` for masked state inspection.
+- **55 → 74 proxy-config tests** — covers both transports, the `apply_env()` independent-toggle guarantee, the single-transport selection logic, and the masked `--show` output.
+
+---
+
+## What's New in v1.5.0
+
+- 37 new entity types across crypto, credentials, messaging, and network/forensic indicators.
+- YARA, Snort, Suricata, and IOC package ZIP exports.
+- Persistent actor profiles with aliases, infrastructure, notes, and timelines.
+- Cross-alias resolution using shared infrastructure, shared PGP, string similarity, temporal co-activity, and co-investigation.
+- Backend graph community detection and path-between-nodes queries.
+- CLI graph browser path finder and frontend Find Path highlighting.
+- Per-phase pipeline timeouts for enrichment, graph, summary, finalize, and parallel sources.
+- `sources_used` and `infrastructure_clusters` persist in investigation metadata.
+- Cross-run enrichment cache with Redis, SQLite, and memory backends.
+- Auto-discovery and weekly validation of `.onion` seeds.
+
+---
+
 ## Quick Start
 
 ### Option A - CLI (no Docker, 30 seconds)
@@ -102,11 +131,28 @@ The Docker stack includes PostgreSQL, Tor, FastAPI, and Next.js.
 |---|---|
 | `voidaccess investigate` | Run an investigation |
 | `voidaccess show` | Interactive entity browser |
-| `voidaccess export` | Export STIX/MISP/Sigma/CSV/MD |
+| `voidaccess export` | Export STIX/MISP/Sigma/YARA/Snort/Suricata/package/CSV/MD/JSON |
+| `voidaccess package <file>` | Export an IOC ZIP bundle |
 | `voidaccess enrich` | Re-enrich saved results |
 | `voidaccess list` | List saved investigations |
-| `voidaccess status` | Config and API key status |
+| `voidaccess status` | Config, API key, cache, engine, and seed status |
+| `voidaccess actors` | List persistent actor profiles |
+| `voidaccess actor <handle>` | Show an actor profile with aliases, infrastructure, notes, and history |
+| `voidaccess actor <handle> --timeline` | Show an actor activity timeline |
+| `voidaccess actor <handle> --note "text"` | Append an analyst note to an actor profile |
+| `voidaccess timeline <handle>` | Shortcut for `voidaccess actor <handle> --timeline` |
 | `voidaccess configure` | Setup wizard |
+| `voidaccess configure proxy` | ScrapingAnt key, username, type, and routing toggles. Flags: `--enable / --disable` (API gate), `--enable-proxy / --disable-proxy` (proxy gate), `--show` (masked state) |
+
+Export examples:
+
+```bash
+voidaccess package investigation.json
+voidaccess export investigation.json --format yara
+voidaccess export investigation.json --format snort
+voidaccess export investigation.json --format suricata
+voidaccess status --seeds
+```
 
 ### CLI vs Docker
 
@@ -167,14 +213,17 @@ VoidAccess handles the complexity of dark web research through a rigorous sequen
 
 ## What It Extracts
 
-The extraction pipeline identifies these entity types:
+The extraction pipeline identifies 55+ entity types:
 
 | Category | Examples |
 |---|---|
-| **Cryptocurrency** | Bitcoin, Ethereum, Monero wallet addresses |
+| **Cryptocurrency** | Bitcoin, Ethereum, Monero, Litecoin, Zcash, Dogecoin, XRP, Solana, Tron, Bitcoin Cash, Dash, ENS |
 | **Network Indicators** | IPv4 addresses, .onion URLs, domains, email addresses, PGP keys |
 | **File Indicators** | MD5, SHA1, SHA256 hashes |
-| **Vulnerabilities** | CVE numbers, MITRE ATT&CK techniques |
+| **Credentials** | AWS keys, GitHub tokens, Slack tokens, Discord tokens, JWTs, Google API keys, Stripe keys, generic API keys, stealer log entries |
+| **Messaging Handles** | Telegram, Discord, XMPP, Tox, Session, Matrix, Wire, ICQ, Wickr |
+| **Network/Forensic** | IPv6, MAC addresses, IPFS CIDs, combo-list entries, YARA rules, MITRE tactics, Exploit-DB IDs, Nuclei templates, seed phrases |
+| **Vulnerabilities** | CVE numbers, MITRE ATT&CK techniques and tactics |
 | **Threat Actors** | Actor handles, malware families, ransomware group names |
 | **Paste Sites** | Pastebin, Ghostbin, Rentry, and similar links |
 | **People/Orgs** | Named persons, organization names, locations |
@@ -214,7 +263,19 @@ Export formats:
 - **STIX 2.1** — bundles with indicators, threat actors, malware objects
 - **MISP JSON** — events with galaxies for direct import
 - **Sigma rules** — auto-generated detection rules from extracted IOCs
-- **CSV** — flat entity dumps for spreadsheet analysis
+- **YARA rules** - generated rules for malware, credentials, infrastructure, and IOC strings
+- **Snort rules** - network detection rules for IPs, domains, URLs, and selected IOC content
+- **Suricata rules** - Suricata-compatible network rules with the same IOC coverage as Snort
+- **IOC package ZIP** - 21-file bundle containing text IOC lists, STIX, MISP, Sigma, YARA, Snort, Suricata, summary, and CSV
+- **CSV** - flat entity dumps for spreadsheet analysis
+
+---
+
+## Actor Intelligence
+
+VoidAccess v1.5.0 persists actor profiles across investigations in `actor_profiles`, with linked aliases and infrastructure in `actor_aliases` and `actor_infrastructure`. Profiles are populated from threat actor, ransomware group, and handle entities, then enriched with co-occurring infrastructure and timeline events.
+
+Cross-alias resolution scores five signals: shared infrastructure, shared PGP, string similarity, temporal co-activity, and co-investigation. Use `voidaccess actors` to list profiles, `voidaccess actor <handle>` for the full profile, `voidaccess actor <handle> --timeline` for chronology, and `voidaccess actor <handle> --note "text"` for analyst notes.
 
 ---
 
@@ -253,6 +314,46 @@ All enrichment sources that require a key degrade gracefully when the key is abs
 | `GITLAB_TOKEN` | Raises GitLab scraping from 15 to 60 req/min | Free | [gitlab.com/profile/personal_access_tokens](https://gitlab.com/-/profile/personal_access_tokens) |
 | `BLOCKCYPHER_TOKEN` | BTC/ETH wallet balance and transaction graph | Yes | [blockcypher.com](https://www.blockcypher.com) |
 | `ETHERSCAN_API_KEY` | ETH wallet lookups | Yes | [etherscan.io/apis](https://etherscan.io/apis) |
+| `SCRAPINGANT_API_KEY` + `VOIDACCESS_USE_PROXIES=true` | Optional clearnet proxy for paste sites + RSS feeds (see below) | Yes (free tier) | [scrapingant.com](https://scrapingant.com/?ref=mzliyzh) |
+
+### Optional: Clearnet Scraping Proxy (ScrapingAnt)
+
+When a third-party clearnet site rate-limits or blocks VoidAccess's outbound IP, every paste site fetch and every RSS feed fetch in the same investigation can fail. The optional **ScrapingAnt** integration routes those specific requests through ScrapingAnt — either its Web Scraping API or its Proxy Mode HTTP CONNECT endpoint at `proxy.scrapingant.com:8080`.
+
+**What it covers** — paste sites (Pastebin, dpaste, paste.ee, Rentry) and the 20 curated RSS security feeds (Krebs on Security, BleepingComputer, Talos, Mandiant, CrowdStrike, Unit 42, CISA, and others). Nothing else.
+
+**What it does not cover** — Tor traffic, dark web scraping, and `.onion` fetches are **completely unaffected** by this setting regardless of how it is configured. The proxy only sees the two clearnet sources named above.
+
+**GitHub and GitLab scraping are also unaffected** — and intentionally so. Both of those scrapers carry authentication tokens (`GITHUB_TOKEN`, `GITLAB_TOKEN`) in their requests. Forwarding those tokens through a third-party proxy would expose them to that third party, which is unacceptable from a security standpoint. Both scrapers always go direct to the GitHub/GitLab API regardless of the proxy setting. This is a permanent design constraint, not something the proxy toggle can override.
+
+**It's entirely optional.** VoidAccess behaves identically without it — paste sites and RSS feeds are simply fetched directly, exactly as they were in every prior release. Add the key only if you see upstream rate-limiting or blocks affecting those two sources.
+
+#### Two mutually exclusive transports (v1.6.0)
+
+Per the [ScrapingAnt docs](https://docs.scrapingant.com/proxy-mode): *"The proxy mode is a light front-end for the scraping API and has all the same functionality and performance as sending requests to the API endpoint."* Therefore the two transports below are **alternate transports to the same backend service** — pick ONE per request, never both:
+
+| Transport | Env var | Required config | What it does |
+|---|---|---|---|
+| **REST API** | `VOIDACCESS_USE_PROXIES=true` | `SCRAPINGANT_API_KEY` | POSTs the target URL to `api.scrapingant.com/v2/general` and returns the response body. Legacy v1.5.0 toggle. |
+| **Proxy Mode** | `VOIDACCESS_USE_PROXY=true` | `SCRAPINGANT_API_KEY` (only) | Routes the request as HTTP CONNECT through `proxy.scrapingant.com:8080` with username string built at connection time per docs: `scrapingant&browser=false&proxy_type=residential\|datacenter`. |
+
+The Proxy Mode transport also reads `SCRAPINGANT_PROXY_TYPE` to pick the pool: `residential` (default; harder to detect, slightly higher latency) or `datacenter` (faster, cheaper, easier to fingerprint).
+
+**Missing credentials leave both transports inactive.** Setting either transport env var to `true` without `SCRAPINGANT_API_KEY` is a no-op for that transport. No errors, no surprises.
+
+**If both transport env vars are set, Proxy Mode wins** with a one-shot info log at runtime — there is no chained mode (Proxy Mode is documented as "the same functionality" as the REST API, so stacking them would double-charge without adding capability).
+
+**How to turn it on** — all four surfaces, covering either transport:
+
+| Surface | How |
+|---|---|
+| CLI configure wizard | `voidaccess configure` then `voidaccess configure keys` — paste sites and RSS feeds will be flagged with their honest "never Tor" description before any field is asked for. The interactive prompt covers the key, pool type, and asks about each transport separately. |
+| `voidaccess configure proxy` (subcommand) | Interactive prompt for key + pool type. Non-interactive flags: `--enable / --disable` (REST API transport), `--enable-proxy / --disable-proxy` (Proxy Mode transport), `--show` (prints masked key `abcd…5678`, pool type, and both transport states). |
+| `setup.sh` during Docker install | Group F in the Enrichment Keys step; prompts for key + pool type, asks about each transport toggle separately. |
+| `--use-proxies` flag (single run) | `voidaccess investigate "query" --use-proxies` — sets `VOIDACCESS_USE_PROXIES=true` (REST API transport) for the current process only, leaves the on-disk config untouched. |
+| Docker / web settings page | Settings → API Keys → ScrapingAnt. Stored encrypted at rest via the existing per-user `UserApiKey` mechanism (Fernet AES-128). |
+
+**Referral signup:** [https://scrapingant.com/?ref=mzliyzh](https://scrapingant.com/?ref=mzliyzh) (referral bonus applied on first paid plan; a free tier is available for low-volume use).
 
 ---
 

{voidaccess-1.4.7 → voidaccess-1.6.0}/README.md

@@ -19,6 +19,35 @@ Commercial threat intelligence platforms often charge prohibitive annual fees fo
 
 ---
 
+## What's New in v1.6.0
+
+- **Optional clearnet scraping proxy (ScrapingAnt)** — paste sites and RSS feeds can now be routed through ScrapingAnt. Affects clearnet scraping only; Tor, `.onion`, GitHub, and GitLab traffic are never affected.
+- **Two mutually exclusive transports** — pick one, not both:
+  - **REST API transport** — `VOIDACCESS_USE_PROXIES=true` (legacy v1.5.0 toggle) routes requests through ScrapingAnt's Web Scraping API.
+  - **Proxy Mode transport** — `VOIDACCESS_USE_PROXY=true` routes requests through ScrapingAnt's HTTP CONNECT endpoint at `proxy.scrapingant.com:8080`.
+  - Per [ScrapingAnt docs](https://docs.scrapingant.com/proxy-mode): "Proxy Mode is a light front-end for the scraping API and has all the same functionality and performance" — so the two are alternate transports to the same backend, never chained.
+- **`SCRAPINGANT_PROXY_TYPE`** — `residential` (default) or `datacenter`; per docs this is passed as a `proxy_type=` parameter in the Proxy Mode username string (which is built at connection time as `"scrapingant&browser=false&proxy_type=..."`). NOT a separate hostname.
+- **Single credential** — `SCRAPINGANT_API_KEY` is the only real credential; the Proxy Mode username is a literal constant per docs. No per-customer username field, no second key.
+- **New CLI surfaces** — `voidaccess configure proxy` now prompts for key + type in one uninterrupted block, plus `--enable-proxy / --disable-proxy` for non-interactive Proxy Mode toggling and `--show` for masked state inspection.
+- **55 → 74 proxy-config tests** — covers both transports, the `apply_env()` independent-toggle guarantee, the single-transport selection logic, and the masked `--show` output.
+
+---
+
+## What's New in v1.5.0
+
+- 37 new entity types across crypto, credentials, messaging, and network/forensic indicators.
+- YARA, Snort, Suricata, and IOC package ZIP exports.
+- Persistent actor profiles with aliases, infrastructure, notes, and timelines.
+- Cross-alias resolution using shared infrastructure, shared PGP, string similarity, temporal co-activity, and co-investigation.
+- Backend graph community detection and path-between-nodes queries.
+- CLI graph browser path finder and frontend Find Path highlighting.
+- Per-phase pipeline timeouts for enrichment, graph, summary, finalize, and parallel sources.
+- `sources_used` and `infrastructure_clusters` persist in investigation metadata.
+- Cross-run enrichment cache with Redis, SQLite, and memory backends.
+- Auto-discovery and weekly validation of `.onion` seeds.
+
+---
+
 ## Quick Start
 
 ### Option A - CLI (no Docker, 30 seconds)
@@ -56,11 +85,28 @@ The Docker stack includes PostgreSQL, Tor, FastAPI, and Next.js.
 |---|---|
 | `voidaccess investigate` | Run an investigation |
 | `voidaccess show` | Interactive entity browser |
-| `voidaccess export` | Export STIX/MISP/Sigma/CSV/MD |
+| `voidaccess export` | Export STIX/MISP/Sigma/YARA/Snort/Suricata/package/CSV/MD/JSON |
+| `voidaccess package <file>` | Export an IOC ZIP bundle |
 | `voidaccess enrich` | Re-enrich saved results |
 | `voidaccess list` | List saved investigations |
-| `voidaccess status` | Config and API key status |
+| `voidaccess status` | Config, API key, cache, engine, and seed status |
+| `voidaccess actors` | List persistent actor profiles |
+| `voidaccess actor <handle>` | Show an actor profile with aliases, infrastructure, notes, and history |
+| `voidaccess actor <handle> --timeline` | Show an actor activity timeline |
+| `voidaccess actor <handle> --note "text"` | Append an analyst note to an actor profile |
+| `voidaccess timeline <handle>` | Shortcut for `voidaccess actor <handle> --timeline` |
 | `voidaccess configure` | Setup wizard |
+| `voidaccess configure proxy` | ScrapingAnt key, username, type, and routing toggles. Flags: `--enable / --disable` (API gate), `--enable-proxy / --disable-proxy` (proxy gate), `--show` (masked state) |
+
+Export examples:
+
+```bash
+voidaccess package investigation.json
+voidaccess export investigation.json --format yara
+voidaccess export investigation.json --format snort
+voidaccess export investigation.json --format suricata
+voidaccess status --seeds
+```
 
 ### CLI vs Docker
 
@@ -121,14 +167,17 @@ VoidAccess handles the complexity of dark web research through a rigorous sequen
 
 ## What It Extracts
 
-The extraction pipeline identifies these entity types:
+The extraction pipeline identifies 55+ entity types:
 
 | Category | Examples |
 |---|---|
-| **Cryptocurrency** | Bitcoin, Ethereum, Monero wallet addresses |
+| **Cryptocurrency** | Bitcoin, Ethereum, Monero, Litecoin, Zcash, Dogecoin, XRP, Solana, Tron, Bitcoin Cash, Dash, ENS |
 | **Network Indicators** | IPv4 addresses, .onion URLs, domains, email addresses, PGP keys |
 | **File Indicators** | MD5, SHA1, SHA256 hashes |
-| **Vulnerabilities** | CVE numbers, MITRE ATT&CK techniques |
+| **Credentials** | AWS keys, GitHub tokens, Slack tokens, Discord tokens, JWTs, Google API keys, Stripe keys, generic API keys, stealer log entries |
+| **Messaging Handles** | Telegram, Discord, XMPP, Tox, Session, Matrix, Wire, ICQ, Wickr |
+| **Network/Forensic** | IPv6, MAC addresses, IPFS CIDs, combo-list entries, YARA rules, MITRE tactics, Exploit-DB IDs, Nuclei templates, seed phrases |
+| **Vulnerabilities** | CVE numbers, MITRE ATT&CK techniques and tactics |
 | **Threat Actors** | Actor handles, malware families, ransomware group names |
 | **Paste Sites** | Pastebin, Ghostbin, Rentry, and similar links |
 | **People/Orgs** | Named persons, organization names, locations |
@@ -168,7 +217,19 @@ Export formats:
 - **STIX 2.1** — bundles with indicators, threat actors, malware objects
 - **MISP JSON** — events with galaxies for direct import
 - **Sigma rules** — auto-generated detection rules from extracted IOCs
-- **CSV** — flat entity dumps for spreadsheet analysis
+- **YARA rules** - generated rules for malware, credentials, infrastructure, and IOC strings
+- **Snort rules** - network detection rules for IPs, domains, URLs, and selected IOC content
+- **Suricata rules** - Suricata-compatible network rules with the same IOC coverage as Snort
+- **IOC package ZIP** - 21-file bundle containing text IOC lists, STIX, MISP, Sigma, YARA, Snort, Suricata, summary, and CSV
+- **CSV** - flat entity dumps for spreadsheet analysis
+
+---
+
+## Actor Intelligence
+
+VoidAccess v1.5.0 persists actor profiles across investigations in `actor_profiles`, with linked aliases and infrastructure in `actor_aliases` and `actor_infrastructure`. Profiles are populated from threat actor, ransomware group, and handle entities, then enriched with co-occurring infrastructure and timeline events.
+
+Cross-alias resolution scores five signals: shared infrastructure, shared PGP, string similarity, temporal co-activity, and co-investigation. Use `voidaccess actors` to list profiles, `voidaccess actor <handle>` for the full profile, `voidaccess actor <handle> --timeline` for chronology, and `voidaccess actor <handle> --note "text"` for analyst notes.
 
 ---
 
@@ -207,6 +268,46 @@ All enrichment sources that require a key degrade gracefully when the key is abs
 | `GITLAB_TOKEN` | Raises GitLab scraping from 15 to 60 req/min | Free | [gitlab.com/profile/personal_access_tokens](https://gitlab.com/-/profile/personal_access_tokens) |
 | `BLOCKCYPHER_TOKEN` | BTC/ETH wallet balance and transaction graph | Yes | [blockcypher.com](https://www.blockcypher.com) |
 | `ETHERSCAN_API_KEY` | ETH wallet lookups | Yes | [etherscan.io/apis](https://etherscan.io/apis) |
+| `SCRAPINGANT_API_KEY` + `VOIDACCESS_USE_PROXIES=true` | Optional clearnet proxy for paste sites + RSS feeds (see below) | Yes (free tier) | [scrapingant.com](https://scrapingant.com/?ref=mzliyzh) |
+
+### Optional: Clearnet Scraping Proxy (ScrapingAnt)
+
+When a third-party clearnet site rate-limits or blocks VoidAccess's outbound IP, every paste site fetch and every RSS feed fetch in the same investigation can fail. The optional **ScrapingAnt** integration routes those specific requests through ScrapingAnt — either its Web Scraping API or its Proxy Mode HTTP CONNECT endpoint at `proxy.scrapingant.com:8080`.
+
+**What it covers** — paste sites (Pastebin, dpaste, paste.ee, Rentry) and the 20 curated RSS security feeds (Krebs on Security, BleepingComputer, Talos, Mandiant, CrowdStrike, Unit 42, CISA, and others). Nothing else.
+
+**What it does not cover** — Tor traffic, dark web scraping, and `.onion` fetches are **completely unaffected** by this setting regardless of how it is configured. The proxy only sees the two clearnet sources named above.
+
+**GitHub and GitLab scraping are also unaffected** — and intentionally so. Both of those scrapers carry authentication tokens (`GITHUB_TOKEN`, `GITLAB_TOKEN`) in their requests. Forwarding those tokens through a third-party proxy would expose them to that third party, which is unacceptable from a security standpoint. Both scrapers always go direct to the GitHub/GitLab API regardless of the proxy setting. This is a permanent design constraint, not something the proxy toggle can override.
+
+**It's entirely optional.** VoidAccess behaves identically without it — paste sites and RSS feeds are simply fetched directly, exactly as they were in every prior release. Add the key only if you see upstream rate-limiting or blocks affecting those two sources.
+
+#### Two mutually exclusive transports (v1.6.0)
+
+Per the [ScrapingAnt docs](https://docs.scrapingant.com/proxy-mode): *"The proxy mode is a light front-end for the scraping API and has all the same functionality and performance as sending requests to the API endpoint."* Therefore the two transports below are **alternate transports to the same backend service** — pick ONE per request, never both:
+
+| Transport | Env var | Required config | What it does |
+|---|---|---|---|
+| **REST API** | `VOIDACCESS_USE_PROXIES=true` | `SCRAPINGANT_API_KEY` | POSTs the target URL to `api.scrapingant.com/v2/general` and returns the response body. Legacy v1.5.0 toggle. |
+| **Proxy Mode** | `VOIDACCESS_USE_PROXY=true` | `SCRAPINGANT_API_KEY` (only) | Routes the request as HTTP CONNECT through `proxy.scrapingant.com:8080` with username string built at connection time per docs: `scrapingant&browser=false&proxy_type=residential\|datacenter`. |
+
+The Proxy Mode transport also reads `SCRAPINGANT_PROXY_TYPE` to pick the pool: `residential` (default; harder to detect, slightly higher latency) or `datacenter` (faster, cheaper, easier to fingerprint).
+
+**Missing credentials leave both transports inactive.** Setting either transport env var to `true` without `SCRAPINGANT_API_KEY` is a no-op for that transport. No errors, no surprises.
+
+**If both transport env vars are set, Proxy Mode wins** with a one-shot info log at runtime — there is no chained mode (Proxy Mode is documented as "the same functionality" as the REST API, so stacking them would double-charge without adding capability).
+
+**How to turn it on** — all four surfaces, covering either transport:
+
+| Surface | How |
+|---|---|
+| CLI configure wizard | `voidaccess configure` then `voidaccess configure keys` — paste sites and RSS feeds will be flagged with their honest "never Tor" description before any field is asked for. The interactive prompt covers the key, pool type, and asks about each transport separately. |
+| `voidaccess configure proxy` (subcommand) | Interactive prompt for key + pool type. Non-interactive flags: `--enable / --disable` (REST API transport), `--enable-proxy / --disable-proxy` (Proxy Mode transport), `--show` (prints masked key `abcd…5678`, pool type, and both transport states). |
+| `setup.sh` during Docker install | Group F in the Enrichment Keys step; prompts for key + pool type, asks about each transport toggle separately. |
+| `--use-proxies` flag (single run) | `voidaccess investigate "query" --use-proxies` — sets `VOIDACCESS_USE_PROXIES=true` (REST API transport) for the current process only, leaves the on-disk config untouched. |
+| Docker / web settings page | Settings → API Keys → ScrapingAnt. Stored encrypted at rest via the existing per-user `UserApiKey` mechanism (Fernet AES-128). |
+
+**Referral signup:** [https://scrapingant.com/?ref=mzliyzh](https://scrapingant.com/?ref=mzliyzh) (referral bonus applied on first paid plan; a free tier is available for low-volume use).
 
 ---
 

{voidaccess-1.4.7 → voidaccess-1.6.0}/api/main.py

@@ -14,7 +14,8 @@ import asyncio
 import logging
 import os
 from contextlib import asynccontextmanager
-from typing import Callable
+from datetime import datetime, timedelta, timezone
+from typing import Callable, Optional
 
 from fastapi import FastAPI, Depends, Request
 from fastapi.exceptions import RequestValidationError
@@ -24,7 +25,7 @@ from slowapi import Limiter
 from slowapi.errors import RateLimitExceeded
 from slowapi.util import get_remote_address
 
-from api.routes import entities, export, investigations, monitors, search, auth, admin, settings
+from api.routes import entities, export, investigations, monitors, search, auth, admin, settings, actors
 from api.auth import get_current_user
 from monitor.scheduler import start_scheduler
 
@@ -118,18 +119,19 @@ async def lifespan(app: FastAPI):
     except Exception as e:
         logger.warning(f"Seed database load failed (non-fatal): {e}")
 
-    # Recover stranded processing investigations
+    # Recover stranded processing investigations (Phase 6.3 startup sweep)
+    # On startup: every investigation left in 'processing' by a previous
+    # process is marked failed — the pipeline tasks that owned them are
+    # gone.  A periodic sweep (every 5 min) handles investigations that
+    # get stuck while the server is alive.
     try:
         if os.getenv("DATABASE_URL"):
-            from db.session import get_session
-            from db.models import Investigation
-            with get_session() as session:
-                stranded_count = session.query(Investigation).filter(Investigation.status == "processing").update(
-                    {"status": "failed", "summary": "Investigation interrupted due to server restart."}
+            swept = await _sweep_stuck_investigations(cutoff_minutes=None)
+            if swept:
+                logger.warning(
+                    "Recovered %d stranded investigations (marked as failed).",
+                    swept,
                 )
-                if stranded_count > 0:
-                    session.commit()
-                    logger.warning(f"Recovered {stranded_count} stranded investigations (marked as failed).")
     except Exception as e:
         logger.warning(f"Failed to recover stranded investigations: {e}")
 
@@ -145,9 +147,31 @@ async def lifespan(app: FastAPI):
         logger.error(f"APScheduler failed to start: {e}")
         scheduler = None
 
+    # Start periodic stuck-investigation sweeper (Phase 6.3). Runs every
+    # 5 minutes and marks investigations stuck in 'processing' for more
+    # than INVESTIGATION_HARD_TIMEOUT_MINUTES as 'failed'.  Cancelled on
+    # shutdown below.
+    _periodic_sweep_task: Optional[asyncio.Task] = None
+    if os.getenv("DATABASE_URL"):
+        try:
+            _periodic_sweep_task = asyncio.create_task(
+                _periodic_stuck_sweep(),
+                name="voidaccess-stuck-investigation-sweeper",
+            )
+            logger.info("Periodic stuck-investigation sweeper started (every 5 min).")
+        except Exception as e:
+            logger.warning(f"Failed to start periodic sweeper: {e}")
+
     yield
 
     # --- Shutdown ---
+    if _periodic_sweep_task is not None and not _periodic_sweep_task.done():
+        _periodic_sweep_task.cancel()
+        try:
+            await _periodic_sweep_task
+        except (asyncio.CancelledError, Exception):
+            pass
+
     if scheduler and scheduler.running:
         scheduler.shutdown(wait=False)
         logger.warning("APScheduler stopped")
@@ -170,6 +194,120 @@ async def lifespan(app: FastAPI):
         pass
 
 
+# ---------------------------------------------------------------------------
+# Stuck-investigation sweeper (Phase 6.3)
+# ---------------------------------------------------------------------------
+# FastAPI BackgroundTasks runs in the same process as the HTTP handler.
+# If the worker crashes mid-investigation, the row stays at status='processing'
+# forever.  This sweeper marks them 'failed' on two schedules:
+#
+#   1. Startup  — cutoff_minutes=None → every 'processing' row is swept
+#                 (the prior process is gone, no legitimate owner).
+#   2. Periodic — every 5 minutes, cutoff = INVESTIGATION_HARD_TIMEOUT_MINUTES
+#                 (configurable via env).  Defends against in-process hangs.
+#
+# The sweep only ever UPDATES status; it never deletes rows.
+
+# Hard timeout after which an investigation is considered permanently stuck.
+# Default 30 min — generous enough to cover the slowest legitimate run
+# (parallel_sources 300s + enrichment 120s + graph 60s + summary 90s + finalize
+# 30s ≈ 10 min on a healthy host; 30 min is 3x that to absorb transient
+# network slowness without false positives).
+INVESTIGATION_HARD_TIMEOUT_MINUTES = int(
+    os.getenv("VOIDACCESS_INVESTIGATION_HARD_TIMEOUT_MINUTES", "30") or 30
+)
+# Periodic sweep interval. 5 min is a good default — catches stuck rows
+# quickly without flooding the DB.
+SWEEP_INTERVAL_SECONDS = int(
+    os.getenv("VOIDACCESS_SWEEP_INTERVAL_SECONDS", "300") or 300
+)
+
+
+async def _sweep_stuck_investigations(cutoff_minutes: Optional[int] = 30) -> int:
+    """Mark investigations stuck in 'processing' as 'failed'.
+
+    Args:
+        cutoff_minutes: Only sweep rows older than this many minutes.
+            ``None`` → startup mode: sweep *all* processing rows (the prior
+            process is gone, no legitimate owner remains).
+            ``int``  → periodic mode: sweep only rows older than the cutoff.
+
+    Returns the number of rows swept.  Returns 0 when DB is unconfigured,
+    the table is missing, or no rows match — never raises.
+    """
+    if not os.getenv("DATABASE_URL"):
+        return 0
+    try:
+        from db.session import get_session
+        from db.models import Investigation
+
+        # Build the query in a short-lived session, do the UPDATE in another.
+        with get_session() as session:
+            query = session.query(Investigation).filter(
+                Investigation.status == "processing"
+            )
+            if cutoff_minutes is not None:
+                cutoff_dt = datetime.now(timezone.utc) - timedelta(
+                    minutes=cutoff_minutes
+                )
+                query = query.filter(Investigation.created_at < cutoff_dt)
+
+            stuck = query.all()
+            if not stuck:
+                return 0
+
+            swept_ids = [inv.id for inv in stuck]
+            sweep_reason = (
+                "Server restarted mid-investigation"
+                if cutoff_minutes is None
+                else f"Investigation timed out after {cutoff_minutes} min — "
+                     "server may have restarted or pipeline may be hung"
+            )
+
+        # Update outside the read session.
+        from sqlalchemy import update
+        with get_session() as session:
+            session.execute(
+                update(Investigation)
+                .where(Investigation.id.in_(swept_ids))
+                .values(
+                    status="failed",
+                    summary=sweep_reason,
+                )
+            )
+            session.commit()
+
+        for inv_id in swept_ids:
+            logger.warning("Swept stuck investigation: %s", inv_id)
+        logger.info("Swept %d stuck investigations (cutoff=%s)", len(swept_ids), cutoff_minutes)
+        return len(swept_ids)
+    except Exception as exc:
+        logger.warning("Swept-investigation sweep failed: %s", exc)
+        return 0
+
+
+async def _periodic_stuck_sweep() -> None:
+    """Background task: every SWEEP_INTERVAL_SECONDS, sweep stuck rows.
+
+    Runs until cancelled by the lifespan teardown.  Sleeps in a loop so
+    cancelling the task is the only stop signal — never raises.
+    """
+    try:
+        while True:
+            await asyncio.sleep(SWEEP_INTERVAL_SECONDS)
+            try:
+                await _sweep_stuck_investigations(
+                    cutoff_minutes=INVESTIGATION_HARD_TIMEOUT_MINUTES,
+                )
+            except asyncio.CancelledError:
+                raise
+            except Exception as exc:
+                logger.warning("Periodic stuck-investigation sweep iteration failed: %s", exc)
+    except asyncio.CancelledError:
+        logger.info("Periodic stuck-investigation sweep cancelled.")
+        return
+
+
 # ---------------------------------------------------------------------------
 # App setup
 # ---------------------------------------------------------------------------
@@ -294,6 +432,12 @@ app.include_router(
     tags=["monitors"],
     dependencies=[Depends(get_current_user)],
 )
+app.include_router(
+    actors.router,
+    prefix="/actors",
+    tags=["actors"],
+    dependencies=[Depends(get_current_user)],
+)
 app.include_router(
     admin.router,
     prefix="/admin",