scraperecon 0.1.0__tar.gz

scraperecon-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,34 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  pypi-publish:
+    name: Build and publish to PyPI
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/scraperecon
+    permissions:
+      id-token: write  # IMPORTANT: mandatory for trusted publishing
+      contents: read   # Needed for checkout
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install pypa/build
+        run: python -m pip install --upgrade pip build
+
+      - name: Build distribution
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

scraperecon-0.1.0/.gitignore

@@ -0,0 +1,90 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# IDEs and Editors
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+.project
+.pydevproject
+
+# OS generated files
+.DS_Store
+.DS_Store?
+._*
+.Spotlight-V100
+.Trashes
+ehthumbs.db
+Thumbs.db
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.pyre/
+
+# Pyre type checker
+.pyre/
+
+# ruff / linters
+.ruff_cache/

scraperecon-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shaheer Sarfaraz
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

scraperecon-0.1.0/PKG-INFO

@@ -0,0 +1,150 @@
+Metadata-Version: 2.4
+Name: scraperecon
+Version: 0.1.0
+Summary: A CLI tool that runs a sequential reconnaissance pipeline against a target URL before a developer writes a scraper.
+License-File: LICENSE
+Requires-Python: >=3.11
+Requires-Dist: curl-cffi>=0.7
+Requires-Dist: httpx>=0.27
+Requires-Dist: rich>=13
+Requires-Dist: typer>=0.12
+Description-Content-Type: text/markdown
+
+# scraperecon
+
+Run this before you write a scraper. It tells you what bot protection a site has, whether plain HTTP or TLS impersonation is enough to get through, and how aggressively it rate limits — before you've written a single line of scraper code.
+
+<img width="1117" height="409" alt="image" src="https://github.com/user-attachments/assets/aa7d0670-c612-4cfb-a579-b610b9c04163" />
+
+<br/>
+
+## Usage
+
+```bash
+scraperecon https://target.com
+```
+
+```
+scraperecon — https://target.com
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+Stage 1 — Plain HTTP (httpx, scraper User-Agent)
+  Status:   403 Forbidden
+  Time:     212ms
+  Verdict:  Blocked
+
+Stage 2 — TLS Impersonation (chrome131)
+  Status:   200 OK
+  Time:     389ms
+  Verdict:  Open
+  Note:     TLS fingerprint was the blocker ✓
+
+Stage 3 — Vendor Detection
+  Vendor:     Cloudflare
+  Confidence: High
+  Signals:    cf-ray header, __cf_bm cookie, challenges.cloudflare.com in body
+
+Stage 4 — Rate Limit Probe
+  Skipped (pass --probe-rate to enable)
+
+Recommendation
+  Use curl_cffi with chrome131 TLS profile
+  No CAPTCHA detected at probe volume
+  Proxy rotation not required at low request rates
+```
+
+---
+
+## What it does
+
+scraperecon runs four stages against a URL in order, stopping early where it can.
+
+**Stage 1 — Plain HTTP**
+
+A basic GET with no tricks. If this comes back clean, you don't need anything else — plain `httpx` or `requests` will work fine and you can stop here.
+
+It also checks whether a 200 response is actually real content or a JS challenge page. Cloudflare in particular loves returning 200 with a challenge rather than a 403. scraperecon catches that and marks it `Challenged` instead of lying to you with a green `Open`.
+
+**Stage 2 — TLS Impersonation**
+
+If Stage 1 was blocked or challenged, it retries using `curl_cffi` impersonating Chrome's TLS fingerprint. A lot of bot detection happens at the TLS handshake level — Python's `requests` library has a completely different fingerprint from a real browser, and that alone is enough to get you blocked on many sites before the server has even looked at your headers. If Stage 2 passes where Stage 1 didn't, you know exactly what the fix is.
+
+**Stage 3 — Vendor Detection**
+
+Inspects headers, cookies, and the response body for known signatures and tells you which bot protection vendor is running. This matters because Cloudflare, DataDome, Akamai, and PerimeterX all require different bypass strategies. Knowing which one you're dealing with upfront saves you from trying things that were never going to work.
+
+**Stage 4 — Rate Limit Probe** _(opt-in)_
+
+Fires N requests with configurable concurrency and watches what happens — hard 429s, silent response time degradation, mid-session redirects. Off by default because blasting a site without thinking about it is bad practice. Pass `--probe-rate` when you actually need the data.
+
+---
+
+## Install
+
+```bash
+pipx install scraperecon
+```
+
+---
+
+## Usage
+
+```bash
+scraperecon https://target.com
+scraperecon https://target.com --probe-rate
+scraperecon https://target.com --probe-rate --concurrency 10 --requests 50
+scraperecon https://target.com --impersonate firefox120
+scraperecon https://target.com --save
+scraperecon https://target.com --json | jq .recommendation
+
+| Flag            | Default   | Description                                                                          |
+| --------------- | --------- | ------------------------------------------------------------------------------------ |
+| `--probe-rate`  | off       | Run Stage 4 rate limit probe                                                         |
+| `--concurrency` | 5         | Workers for rate probe                                                               |
+| `--requests`    | 20        | Total requests for rate probe                                                        |
+| `--impersonate` | chrome131 | TLS profile for Stage 2. Options: `chrome131`, `chrome120`, `firefox120`, `safari17` |
+| `--timeout`     | 10        | Per-request timeout in seconds                                                       |
+| `--json`        | off       | Machine-readable JSON output                                                         |
+| `--save`        | off       | Save the full HTML responses to local files (`<domain>_stage1.html`, etc.)           |
+| `--skip-tls`    | off       | Skip Stage 2                                                                         |
+| `--skip-vendor` | off       | Skip Stage 3                                                                         |
+
+---
+
+## Reading the recommendation
+
+At the end of every run you get a plain-English recommendation based on what was found.
+
+- **Plain HTTP should be sufficient** — `httpx` or `requests` will work. No special setup needed.
+- **Use curl_cffi with `<profile>`** — TLS fingerprinting is blocking you. Switch to `curl_cffi` with the listed profile.
+- **May need browser automation** — both plain and TLS requests were blocked. You're likely looking at a full JS challenge (Turnstile, hCaptcha). Playwright with a stealth plugin is probably your next move.
+- **Proxy rotation recommended** — the rate probe hit throttling. At any real request volume you'll need rotating proxies.
+- **CAPTCHA detected** — the response body contained CAPTCHA indicators. Automated solving or a managed scraping service required.
+
+---
+
+## Adding vendor signatures
+
+Signatures live in `scraperecon/data/signatures.json`. It's a flat JSON file — no code required. If you know a signal that's missing, open a PR.
+
+```json
+{
+  "name": "YourVendor",
+  "signals": [
+    { "type": "header_present", "key": "x-your-vendor", "weight": 0.8 },
+    { "type": "cookie_name", "value": "your_cookie", "weight": 0.6 }
+  ]
+}
+```
+
+Signal types: `header_present`, `header_value`, `cookie_name`, `body_contains`, `status_code`.
+
+---
+
+## What it won't do
+
+scraperecon is a recon tool, not a scraping library. It tells you what you need — it doesn't do it for you. No CAPTCHA solving, no Playwright integration, no proxy support, no persistent history.
+
+---
+
+Every scraper project starts with the same 20 minutes of manual work: try curl, get blocked, try curl_cffi, check the headers, fire some requests and see what happens. This automates that.

scraperecon-0.1.0/README.md

@@ -0,0 +1,138 @@
+# scraperecon
+
+Run this before you write a scraper. It tells you what bot protection a site has, whether plain HTTP or TLS impersonation is enough to get through, and how aggressively it rate limits — before you've written a single line of scraper code.
+
+<img width="1117" height="409" alt="image" src="https://github.com/user-attachments/assets/aa7d0670-c612-4cfb-a579-b610b9c04163" />
+
+<br/>
+
+## Usage
+
+```bash
+scraperecon https://target.com
+```
+
+```
+scraperecon — https://target.com
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+Stage 1 — Plain HTTP (httpx, scraper User-Agent)
+  Status:   403 Forbidden
+  Time:     212ms
+  Verdict:  Blocked
+
+Stage 2 — TLS Impersonation (chrome131)
+  Status:   200 OK
+  Time:     389ms
+  Verdict:  Open
+  Note:     TLS fingerprint was the blocker ✓
+
+Stage 3 — Vendor Detection
+  Vendor:     Cloudflare
+  Confidence: High
+  Signals:    cf-ray header, __cf_bm cookie, challenges.cloudflare.com in body
+
+Stage 4 — Rate Limit Probe
+  Skipped (pass --probe-rate to enable)
+
+Recommendation
+  Use curl_cffi with chrome131 TLS profile
+  No CAPTCHA detected at probe volume
+  Proxy rotation not required at low request rates
+```
+
+---
+
+## What it does
+
+scraperecon runs four stages against a URL in order, stopping early where it can.
+
+**Stage 1 — Plain HTTP**
+
+A basic GET with no tricks. If this comes back clean, you don't need anything else — plain `httpx` or `requests` will work fine and you can stop here.
+
+It also checks whether a 200 response is actually real content or a JS challenge page. Cloudflare in particular loves returning 200 with a challenge rather than a 403. scraperecon catches that and marks it `Challenged` instead of lying to you with a green `Open`.
+
+**Stage 2 — TLS Impersonation**
+
+If Stage 1 was blocked or challenged, it retries using `curl_cffi` impersonating Chrome's TLS fingerprint. A lot of bot detection happens at the TLS handshake level — Python's `requests` library has a completely different fingerprint from a real browser, and that alone is enough to get you blocked on many sites before the server has even looked at your headers. If Stage 2 passes where Stage 1 didn't, you know exactly what the fix is.
+
+**Stage 3 — Vendor Detection**
+
+Inspects headers, cookies, and the response body for known signatures and tells you which bot protection vendor is running. This matters because Cloudflare, DataDome, Akamai, and PerimeterX all require different bypass strategies. Knowing which one you're dealing with upfront saves you from trying things that were never going to work.
+
+**Stage 4 — Rate Limit Probe** _(opt-in)_
+
+Fires N requests with configurable concurrency and watches what happens — hard 429s, silent response time degradation, mid-session redirects. Off by default because blasting a site without thinking about it is bad practice. Pass `--probe-rate` when you actually need the data.
+
+---
+
+## Install
+
+```bash
+pipx install scraperecon
+```
+
+---
+
+## Usage
+
+```bash
+scraperecon https://target.com
+scraperecon https://target.com --probe-rate
+scraperecon https://target.com --probe-rate --concurrency 10 --requests 50
+scraperecon https://target.com --impersonate firefox120
+scraperecon https://target.com --save
+scraperecon https://target.com --json | jq .recommendation
+
+| Flag            | Default   | Description                                                                          |
+| --------------- | --------- | ------------------------------------------------------------------------------------ |
+| `--probe-rate`  | off       | Run Stage 4 rate limit probe                                                         |
+| `--concurrency` | 5         | Workers for rate probe                                                               |
+| `--requests`    | 20        | Total requests for rate probe                                                        |
+| `--impersonate` | chrome131 | TLS profile for Stage 2. Options: `chrome131`, `chrome120`, `firefox120`, `safari17` |
+| `--timeout`     | 10        | Per-request timeout in seconds                                                       |
+| `--json`        | off       | Machine-readable JSON output                                                         |
+| `--save`        | off       | Save the full HTML responses to local files (`<domain>_stage1.html`, etc.)           |
+| `--skip-tls`    | off       | Skip Stage 2                                                                         |
+| `--skip-vendor` | off       | Skip Stage 3                                                                         |
+
+---
+
+## Reading the recommendation
+
+At the end of every run you get a plain-English recommendation based on what was found.
+
+- **Plain HTTP should be sufficient** — `httpx` or `requests` will work. No special setup needed.
+- **Use curl_cffi with `<profile>`** — TLS fingerprinting is blocking you. Switch to `curl_cffi` with the listed profile.
+- **May need browser automation** — both plain and TLS requests were blocked. You're likely looking at a full JS challenge (Turnstile, hCaptcha). Playwright with a stealth plugin is probably your next move.
+- **Proxy rotation recommended** — the rate probe hit throttling. At any real request volume you'll need rotating proxies.
+- **CAPTCHA detected** — the response body contained CAPTCHA indicators. Automated solving or a managed scraping service required.
+
+---
+
+## Adding vendor signatures
+
+Signatures live in `scraperecon/data/signatures.json`. It's a flat JSON file — no code required. If you know a signal that's missing, open a PR.
+
+```json
+{
+  "name": "YourVendor",
+  "signals": [
+    { "type": "header_present", "key": "x-your-vendor", "weight": 0.8 },
+    { "type": "cookie_name", "value": "your_cookie", "weight": 0.6 }
+  ]
+}
+```
+
+Signal types: `header_present`, `header_value`, `cookie_name`, `body_contains`, `status_code`.
+
+---
+
+## What it won't do
+
+scraperecon is a recon tool, not a scraping library. It tells you what you need — it doesn't do it for you. No CAPTCHA solving, no Playwright integration, no proxy support, no persistent history.
+
+---
+
+Every scraper project starts with the same 20 minutes of manual work: try curl, get blocked, try curl_cffi, check the headers, fire some requests and see what happens. This automates that.

scraperecon-0.1.0/pyproject.toml

@@ -0,0 +1,19 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "scraperecon"
+version = "0.1.0"
+description = "A CLI tool that runs a sequential reconnaissance pipeline against a target URL before a developer writes a scraper."
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "httpx>=0.27",
+    "curl-cffi>=0.7",
+    "typer>=0.12",
+    "rich>=13"
+]
+
+[project.scripts]
+scraperecon = "scraperecon.main:app"

scraperecon-0.1.0/scraperecon/data/indicators.json

@@ -0,0 +1,14 @@
+[
+  "cf-browser-verification",
+  "challenges.cloudflare.com",
+  "turnstile",
+  "cf_chl",
+  "jschl_vc",
+  "ray id",
+  "enable javascript",
+  "checking your browser",
+  "datadome",
+  "px-captcha",
+  "captcha",
+  "challenge"
+]

scraperecon-0.1.0/scraperecon/data/signatures.json

@@ -0,0 +1,61 @@
+{
+  "vendors": [
+    {
+      "name": "Cloudflare",
+      "signals": [
+        { "type": "header_present", "key": "cf-ray", "weight": 0.6 },
+        { "type": "cookie_name", "value": "__cf_bm", "weight": 0.3 },
+        { "type": "cookie_name", "value": "cf_clearance", "weight": 0.3 },
+        { "type": "header_value", "key": "server", "value": "cloudflare", "weight": 0.2 },
+        { "type": "body_contains", "value": "cf-browser-verification", "weight": 0.4 },
+        { "type": "body_contains", "value": "challenges.cloudflare.com", "weight": 0.5 }
+      ]
+    },
+    {
+      "name": "DataDome",
+      "signals": [
+        { "type": "cookie_name", "value": "datadome", "weight": 0.8 },
+        { "type": "header_present", "key": "x-datadome-cid", "weight": 0.5 },
+        { "type": "body_contains", "value": "datadome", "weight": 0.4 }
+      ]
+    },
+    {
+      "name": "Akamai",
+      "signals": [
+        { "type": "cookie_name", "value": "_abck", "weight": 0.7 },
+        { "type": "cookie_name", "value": "bm_sz", "weight": 0.5 },
+        { "type": "header_present", "key": "x-akamai-edgescape", "weight": 0.4 }
+      ]
+    },
+    {
+      "name": "PerimeterX",
+      "signals": [
+        { "type": "cookie_name", "value": "_px", "weight": 0.6 },
+        { "type": "cookie_name", "value": "_pxhd", "weight": 0.4 },
+        { "type": "body_contains", "value": "px-captcha", "weight": 0.5 }
+      ]
+    },
+    {
+      "name": "Kasada",
+      "signals": [
+        { "type": "header_present", "key": "x-kasada-pow", "weight": 0.8 },
+        { "type": "body_contains", "value": "kasada", "weight": 0.5 }
+      ]
+    },
+    {
+      "name": "Imperva",
+      "signals": [
+        { "type": "cookie_name", "value": "incap_ses", "weight": 0.6 },
+        { "type": "cookie_name", "value": "visid_incap", "weight": 0.5 },
+        { "type": "header_present", "key": "x-iinfo", "weight": 0.4 }
+      ]
+    },
+    {
+      "name": "AWS WAF",
+      "signals": [
+        { "type": "cookie_name", "value": "aws-waf-token", "weight": 0.8 },
+        { "type": "body_contains", "value": "awswaf", "weight": 0.5 }
+      ]
+    }
+  ]
+}

scraperecon-0.1.0/scraperecon/main.py

@@ -0,0 +1,230 @@
+import sys
+import json
+import dataclasses
+from enum import Enum
+import typer
+from rich.console import Console
+from rich.text import Text
+from .pipeline import run_pipeline
+from .types import ReconReport, Verdict, Confidence
+
+app = typer.Typer(add_completion=False)
+
+def _default_json(obj):
+    if isinstance(obj, Enum):
+        return obj.value
+    if dataclasses.is_dataclass(obj):
+        return dataclasses.asdict(obj)
+    raise TypeError(f"Object of type {type(obj)} is not JSON serializable")
+
+def format_verdict(verdict: Verdict) -> Text:
+    if verdict == Verdict.OPEN:
+        return Text("Open", style="bold green")
+    elif verdict == Verdict.BLOCKED:
+        return Text("Blocked", style="bold red")
+    elif verdict == Verdict.CHALLENGED:
+        return Text("Challenged", style="bold magenta")
+    elif verdict in (Verdict.UNCERTAIN, Verdict.SKIPPED):
+        return Text(verdict.value, style="bold yellow")
+    elif verdict == Verdict.ERROR:
+        return Text("Error", style="bold red")
+    else:
+        return Text(verdict.value, style="bold")
+
+def print_human(report: ReconReport):
+    console = Console()
+    err_console = Console(stderr=True)
+    
+    console.print(f"[bold]scraperecon v0.1.0[/bold] — {report.target}")
+    console.print("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
+    console.print()
+    
+    # Stage 1
+    console.print("[bold]Stage 1 — Plain HTTP (httpx, scraper User-Agent)[/bold]")
+    if report.plain.error:
+        err_console.print(f"  [red]Error:[/red] {report.plain.error}")
+        console.print("  Verdict:  ", format_verdict(report.plain.verdict))
+    else:
+        console.print(f"  Status:   {report.plain.status}")
+        console.print(f"  Time:     {report.plain.response_time_ms}ms")
+        console.print("  Verdict:  ", format_verdict(report.plain.verdict))
+    console.print()
+    
+    # Stage 2
+    if report.tls:
+        console.print(f"[bold]Stage 2 — TLS Impersonation ({report.tls.profile_used})[/bold]")
+        if report.tls.error:
+            err_console.print(f"  [red]Error:[/red] {report.tls.error}")
+            console.print("  Verdict:  ", format_verdict(report.tls.verdict))
+        elif report.tls.verdict == Verdict.SKIPPED:
+            console.print("  [yellow]Skipped (Stage 1 was Open)[/yellow]")
+        else:
+            console.print(f"  Status:   {report.tls.status}")
+            console.print(f"  Time:     {report.tls.response_time_ms}ms")
+            console.print("  Verdict:  ", format_verdict(report.tls.verdict))
+            if report.tls.tls_was_blocker:
+                console.print("  Note:     [green]TLS fingerprint was the blocker ✓[/green]")
+    else:
+        console.print("[bold]Stage 2 — TLS Impersonation[/bold]")
+        console.print("  [yellow]Skipped[/yellow]")
+    console.print()
+    
+    # Stage 3
+    console.print("[bold]Stage 3 — Vendor Detection[/bold]")
+    if report.vendor:
+        if report.vendor.vendor:
+            console.print(f"  Vendor:     [bold]{report.vendor.vendor}[/bold]")
+            console.print(f"  Confidence: {report.vendor.confidence.value}")
+            console.print(f"  Signals:    {', '.join(report.vendor.matched_signals)}")
+        else:
+            console.print("  [dim]No known vendor detected[/dim]")
+    else:
+        console.print("  [yellow]Skipped[/yellow]")
+    console.print()
+    
+    # Stage 4
+    console.print("[bold]Stage 4 — Rate Limit Probe[/bold]")
+    if report.rate_limit:
+        console.print(f"  Requests:   {report.rate_limit.successful} successful, {report.rate_limit.blocked} blocked (out of {report.rate_limit.total_requests})")
+        console.print(f"  Median ms:  {report.rate_limit.median_response_ms}ms")
+        if report.rate_limit.block_type:
+            console.print(f"  Block Type: [bold red]{report.rate_limit.block_type.value}[/bold red]")
+            if report.rate_limit.estimated_safe_rps:
+                console.print(f"  Est. Safe:  ~{report.rate_limit.estimated_safe_rps:.1f} req/s")
+        else:
+            console.print("  [green]No rate limit detected at this volume[/green]")
+    else:
+        console.print("  [yellow]Skipped (pass --probe-rate to enable)[/yellow]")
+    console.print()
+    
+    # Recommendations
+    console.print("[bold]Recommendation[/bold]")
+    rec = report.recommendation
+    if rec.use_tls_impersonation:
+        console.print(f"  [green]Use curl_cffi with {rec.profile or 'appropriate'} TLS profile[/green]")
+    else:
+        console.print("  [green]Plain HTTP (httpx/requests) should be sufficient[/green]")
+        
+    if rec.captcha_detected:
+        console.print("  [red]CAPTCHA/Challenge detected[/red]")
+    else:
+        console.print("  No CAPTCHA detected")
+        
+    if rec.proxy_recommended:
+        console.print("  [yellow]Proxy rotation recommended due to rate limits[/yellow]")
+    else:
+        console.print("  Proxy rotation not strictly required at tested volume")
+        
+    for note in rec.notes:
+        console.print(f"  Note: {note}")
+
+def print_json(report: ReconReport):
+    # Map to the requested JSON format
+    out = {
+        "target": report.target,
+        "stages": {
+            "plain": None,
+            "tls": None,
+            "vendor": None,
+            "rate_limit": None
+        },
+        "recommendation": dataclasses.asdict(report.recommendation)
+    }
+    
+    if report.plain:
+        out["stages"]["plain"] = {
+            "verdict": report.plain.verdict.value,
+            "status": report.plain.status,
+            "response_time_ms": report.plain.response_time_ms
+        }
+        if report.plain.error:
+            out["stages"]["plain"]["error"] = report.plain.error
+            
+    if report.tls:
+        out["stages"]["tls"] = {
+            "verdict": report.tls.verdict.value,
+            "status": report.tls.status,
+            "response_time_ms": report.tls.response_time_ms,
+            "tls_was_blocker": report.tls.tls_was_blocker,
+            "profile": report.tls.profile_used
+        }
+        if report.tls.error:
+            out["stages"]["tls"]["error"] = report.tls.error
+            
+    if report.vendor and report.vendor.vendor:
+        out["stages"]["vendor"] = {
+            "vendor": report.vendor.vendor,
+            "confidence": report.vendor.confidence.value if report.vendor.confidence else None,
+            "matched_signals": report.vendor.matched_signals
+        }
+        
+    if report.rate_limit:
+        out["stages"]["rate_limit"] = {
+            "total_requests": report.rate_limit.total_requests,
+            "successful": report.rate_limit.successful,
+            "blocked": report.rate_limit.blocked,
+            "block_type": report.rate_limit.block_type.value if report.rate_limit.block_type else None,
+            "estimated_safe_rps": report.rate_limit.estimated_safe_rps,
+            "retry_after_secs": report.rate_limit.retry_after_secs,
+            "median_response_ms": report.rate_limit.median_response_ms
+        }
+        
+    print(json.dumps(out, indent=2))
+
+@app.command()
+def main(
+    url: str = typer.Argument(..., help="Target URL"),
+    probe_rate: bool = typer.Option(False, "--probe-rate", help="Run stage 4 (rate limit probe)"),
+    concurrency: int = typer.Option(5, "--concurrency", help="Workers for rate probe"),
+    requests: int = typer.Option(20, "--requests", help="Total requests for rate probe"),
+    impersonate: str = typer.Option("chrome131", "--impersonate", help="TLS profile for stage 2"),
+    timeout: int = typer.Option(10, "--timeout", help="Per-request timeout in seconds"),
+    json_out: bool = typer.Option(False, "--json", help="Output machine-readable JSON"),
+    skip_tls: bool = typer.Option(False, "--skip-tls", help="Skip stage 2"),
+    skip_vendor: bool = typer.Option(False, "--skip-vendor", help="Skip stage 3"),
+    save: bool = typer.Option(False, "--save", help="Save the full HTML responses to local files"),
+    version: bool = typer.Option(False, "--version", help="Print version")
+):
+    if version:
+        print("scraperecon v0.1.0")
+        raise typer.Exit()
+        
+    report = run_pipeline(
+        url=url,
+        probe_rate=probe_rate,
+        concurrency=concurrency,
+        requests=requests,
+        impersonate=impersonate,
+        timeout=timeout,
+        skip_tls=skip_tls,
+        skip_vendor=skip_vendor
+    )
+    
+    if json_out:
+        print_json(report)
+    else:
+        print_human(report)
+        
+    if save:
+        from urllib.parse import urlparse
+        domain = urlparse(report.target).netloc or "target"
+        domain = domain.replace(":", "_")
+        
+        console = Console()
+        console.print()
+        console.print("[bold]Saved Files[/bold]")
+        
+        if report.plain and not report.plain.error and report.plain.full_body:
+            fname = f"{domain}_stage1.html"
+            with open(fname, "w", encoding="utf-8") as f:
+                f.write(report.plain.full_body)
+            console.print(f"  [green]Stage 1 saved to:[/green] {fname}")
+            
+        if report.tls and report.tls.verdict not in (Verdict.SKIPPED, Verdict.ERROR) and report.tls.full_body:
+            fname = f"{domain}_stage2.html"
+            with open(fname, "w", encoding="utf-8") as f:
+                f.write(report.tls.full_body)
+            console.print(f"  [green]Stage 2 saved to:[/green] {fname}")
+        
+if __name__ == "__main__":
+    app()

scraperecon-0.1.0/scraperecon/pipeline.py

@@ -0,0 +1,49 @@
+import asyncio
+from .stages import plain, tls, vendor, ratelimit
+from .report import build_recommendation
+from .types import ReconReport
+
+def run_pipeline(
+    url: str,
+    probe_rate: bool,
+    concurrency: int,
+    requests: int,
+    impersonate: str,
+    timeout: int,
+    skip_tls: bool,
+    skip_vendor: bool
+) -> ReconReport:
+
+    # Stage 1
+    plain_res = plain.run(url, timeout)
+
+    # Stage 2
+    if skip_tls or plain_res.error:
+        tls_res = None
+    else:
+        tls_res = tls.run(url, plain_res, impersonate, timeout)
+
+    # Stage 3
+    if skip_vendor:
+        vendor_res = None
+    else:
+        vendor_res = vendor.run(plain_res, tls_res)
+        
+    # Stage 4
+    if probe_rate and ((plain_res and not plain_res.error) or (tls_res and not tls_res.error)):
+        rate_res = asyncio.run(ratelimit.run(
+            url, True, requests, concurrency, plain_res, tls_res, impersonate
+        ))
+    else:
+        rate_res = None
+        
+    rec = build_recommendation(plain_res, tls_res, vendor_res, rate_res)
+    
+    return ReconReport(
+        target=url,
+        plain=plain_res,
+        tls=tls_res,
+        vendor=vendor_res,
+        rate_limit=rate_res,
+        recommendation=rec
+    )

scraperecon-0.1.0/scraperecon/report.py

@@ -0,0 +1,47 @@
+from typing import Optional
+from .types import Recommendation, ReconReport, PlainResult, TlsResult, VendorResult, RateLimitResult, Verdict, BlockType
+
+def build_recommendation(
+    plain: PlainResult,
+    tls: Optional[TlsResult],
+    vendor: Optional[VendorResult],
+    rate: Optional[RateLimitResult]
+) -> Recommendation:
+    
+    use_tls_impersonation = False
+    profile = None
+    captcha_detected = False
+    proxy_recommended = False
+    notes = []
+    
+    if plain.verdict == Verdict.OPEN:
+        use_tls_impersonation = False
+    elif tls and tls.tls_was_blocker:
+        use_tls_impersonation = True
+        profile = tls.profile_used
+    elif plain.verdict in (Verdict.BLOCKED, Verdict.CHALLENGED) and tls and tls.verdict in (Verdict.BLOCKED, Verdict.CHALLENGED):
+        use_tls_impersonation = True
+        notes.append("Both stages blocked/challenged: may need browser automation (Playwright + stealth)")
+        
+    if vendor and vendor.vendor == "Cloudflare":
+        notes.append("Cloudflare detected: consider Playwright + stealth plugin if curl_cffi fails")
+        
+    if rate and rate.block_type is not None:
+        proxy_recommended = True
+        
+    body_to_check = ""
+    if tls and tls.verdict not in (Verdict.SKIPPED, Verdict.ERROR):
+        body_to_check = tls.body_preview.lower()
+    else:
+        body_to_check = plain.body_preview.lower()
+        
+    if "captcha" in body_to_check or "challenge" in body_to_check:
+        captcha_detected = True
+        
+    return Recommendation(
+        use_tls_impersonation=use_tls_impersonation,
+        profile=profile,
+        captcha_detected=captcha_detected,
+        proxy_recommended=proxy_recommended,
+        notes=notes
+    )

scraperecon-0.1.0/scraperecon/signatures.py

@@ -0,0 +1,7 @@
+import json
+import importlib.resources
+from . import data
+
+def load_signatures() -> dict:
+    text = importlib.resources.read_text(data, "signatures.json")
+    return json.loads(text)

scraperecon-0.1.0/scraperecon/stages/plain.py

@@ -0,0 +1,67 @@
+import time
+import httpx
+from ..types import PlainResult, Verdict
+from ..utils import is_challenge_body
+
+DEFAULT_HEADERS = {
+    "User-Agent": "Mozilla/5.0 (compatible; scraperecon/0.1)",
+    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+    "Accept-Language": "en-US,en;q=0.5"
+}
+
+def get_verdict(status: int, body_preview: str = "") -> Verdict:
+    if is_challenge_body(body_preview):
+        return Verdict.CHALLENGED
+    if 200 <= status <= 299:
+        return Verdict.OPEN
+    elif status in (301, 302, 307, 308):
+        return Verdict.REDIRECTED
+    elif status in (403, 429, 503):
+        return Verdict.BLOCKED
+    else:
+        return Verdict.UNCERTAIN
+
+def run(url: str, timeout: int) -> PlainResult:
+    start_time = time.perf_counter()
+    try:
+        with httpx.Client(timeout=timeout, follow_redirects=True) as client:
+            resp = client.get(url, headers=DEFAULT_HEADERS)
+        
+        response_time_ms = int((time.perf_counter() - start_time) * 1000)
+        
+        headers = {k.lower(): v for k, v in resp.headers.items()}
+        cookies = list(resp.cookies.keys())
+        body_preview = resp.text[:2048]
+        final_url = str(resp.url)
+        
+        # Check history to see if there was a redirect
+        if resp.history:
+            # We followed redirects, the final status is what matters for Open/Blocked/etc.
+            # But the spec says "200-299 => OPEN, 301.. => REDIRECTED". If we follow redirects, 
+            # we might just return the final status. Let's return the final status verdict.
+            verdict = get_verdict(resp.status_code, resp.text)
+        else:
+            verdict = get_verdict(resp.status_code, resp.text)
+            
+        return PlainResult(
+            verdict=verdict,
+            status=resp.status_code,
+            response_time_ms=response_time_ms,
+            headers=headers,
+            cookies=cookies,
+            body_preview=body_preview,
+            final_url=final_url,
+            full_body=resp.text
+        )
+    except Exception as e:
+        response_time_ms = int((time.perf_counter() - start_time) * 1000)
+        return PlainResult(
+            verdict=Verdict.ERROR,
+            status=None,
+            response_time_ms=response_time_ms,
+            headers={},
+            cookies=[],
+            body_preview="",
+            final_url=url,
+            error=str(e)
+        )

scraperecon-0.1.0/scraperecon/stages/ratelimit.py

@@ -0,0 +1,146 @@
+import asyncio
+import time
+from typing import Optional
+from curl_cffi import requests
+import httpx
+from ..types import RateLimitResult, BlockType, PlainResult, TlsResult, Verdict
+from .plain import DEFAULT_HEADERS
+
+async def _worker(queue: asyncio.Queue, results: list, client, is_httpx: bool):
+    while True:
+        try:
+            url = queue.get_nowait()
+        except asyncio.QueueEmpty:
+            break
+            
+        start_time = time.perf_counter()
+        try:
+            if is_httpx:
+                resp = await client.get(url, headers=DEFAULT_HEADERS)
+                status = resp.status_code
+                retry_after = resp.headers.get("retry-after")
+                is_redirect = status in (301, 302, 307, 308)
+            else:
+                resp = await client.get(url, headers=DEFAULT_HEADERS, allow_redirects=False)
+                status = resp.status_code
+                retry_after = resp.headers.get("retry-after")
+                is_redirect = status in (301, 302, 307, 308)
+                
+            elapsed = int((time.perf_counter() - start_time) * 1000)
+            
+            is_soft_block = status in (429, 503) or is_redirect
+            
+            results.append({
+                "status": status,
+                "elapsed": elapsed,
+                "is_soft_block": is_soft_block,
+                "retry_after": retry_after
+            })
+        except Exception:
+            elapsed = int((time.perf_counter() - start_time) * 1000)
+            results.append({
+                "status": 0,
+                "elapsed": elapsed,
+                "is_soft_block": False,
+                "retry_after": None
+            })
+            
+        queue.task_done()
+
+async def run(
+    url: str, 
+    probe_rate: bool, 
+    requests_count: int, 
+    concurrency: int, 
+    plain_result: PlainResult, 
+    tls_result: Optional[TlsResult],
+    profile: str
+) -> Optional[RateLimitResult]:
+    if not probe_rate:
+        return None
+
+    use_plain = False
+    if tls_result is None or tls_result.verdict == Verdict.SKIPPED:
+        use_plain = True
+
+    queue = asyncio.Queue()
+    for _ in range(requests_count):
+        queue.put_nowait(url)
+
+    results = []
+    
+    probe_start = time.perf_counter()
+    
+    if use_plain:
+        async with httpx.AsyncClient(verify=False) as client:
+            tasks = []
+            for _ in range(concurrency):
+                tasks.append(asyncio.create_task(_worker(queue, results, client, True)))
+                await asyncio.sleep(0.1) # 100ms jitter
+            await asyncio.gather(*tasks)
+    else:
+        async with requests.AsyncSession(impersonate=profile) as client:
+            tasks = []
+            for _ in range(concurrency):
+                tasks.append(asyncio.create_task(_worker(queue, results, client, False)))
+                await asyncio.sleep(0.1)
+            await asyncio.gather(*tasks)
+
+    probe_duration = time.perf_counter() - probe_start
+
+    successful = sum(1 for r in results if 200 <= r["status"] < 300)
+    blocked = sum(1 for r in results if r["status"] >= 400 or r["status"] == 0)
+    
+    block_type = None
+    retry_after_secs = None
+    
+    for r in results:
+        if r["status"] == 429:
+            block_type = BlockType.RATE_LIMITED
+            if r["retry_after"] and r["retry_after"].isdigit():
+                retry_after_secs = int(r["retry_after"])
+            break
+            
+    if not block_type:
+        for r in results:
+            if r["status"] == 403:
+                block_type = BlockType.HARD_BLOCK
+                break
+                
+    if not block_type:
+        for r in results:
+            if r["is_soft_block"] and r["status"] not in (429, 503):
+                block_type = BlockType.SOFT_REDIRECT
+                break
+                
+    times = sorted(r["elapsed"] for r in results)
+    median_response_ms = times[len(times)//2] if times else 0
+
+    if not block_type and len(results) >= 2:
+        half = len(results) // 2
+        first_half = sorted(r["elapsed"] for r in results[:half])
+        second_half = sorted(r["elapsed"] for r in results[-half:])
+        
+        m1 = first_half[len(first_half)//2] if first_half else 0
+        m2 = second_half[len(second_half)//2] if second_half else 0
+        
+        if m1 > 0 and m2 > m1 * 3:
+            block_type = BlockType.SILENT
+
+    avg_response_time_secs = (sum(r["elapsed"] for r in results) / len(results)) / 1000 if results else 1.0
+    if avg_response_time_secs == 0:
+        avg_response_time_secs = 0.001
+        
+    estimated_safe_rps = None
+    if block_type and requests_count > 0:
+        estimated_safe_rps = (successful / requests_count) * (concurrency / avg_response_time_secs)
+
+    return RateLimitResult(
+        total_requests=requests_count,
+        successful=successful,
+        blocked=blocked,
+        block_type=block_type,
+        estimated_safe_rps=estimated_safe_rps,
+        retry_after_secs=retry_after_secs,
+        median_response_ms=median_response_ms
+    )

scraperecon-0.1.0/scraperecon/stages/tls.py

@@ -0,0 +1,64 @@
+import time
+from curl_cffi import requests
+from ..types import TlsResult, Verdict, PlainResult
+from .plain import DEFAULT_HEADERS, get_verdict
+
+def run(url: str, plain_result: PlainResult, profile: str, timeout: int) -> TlsResult:
+    if plain_result.verdict == Verdict.OPEN:
+        return TlsResult(
+            verdict=Verdict.SKIPPED,
+            status=None,
+            response_time_ms=0,
+            headers={},
+            cookies=[],
+            body_preview="",
+            tls_was_blocker=False,
+            profile_used=profile
+        )
+    
+    start_time = time.perf_counter()
+    try:
+        resp = requests.get(
+            url, 
+            headers=DEFAULT_HEADERS, 
+            impersonate=profile, 
+            timeout=timeout, 
+            allow_redirects=True
+        )
+        
+        response_time_ms = int((time.perf_counter() - start_time) * 1000)
+        
+        headers = {k.lower(): v for k, v in resp.headers.items()}
+        cookies = list(resp.cookies.keys())
+        body_preview = resp.text[:2048]
+        
+        verdict = get_verdict(resp.status_code, body_preview)
+        
+        tls_was_blocker = False
+        if plain_result.verdict in (Verdict.BLOCKED, Verdict.CHALLENGED) and verdict == Verdict.OPEN:
+            tls_was_blocker = True
+            
+        return TlsResult(
+            verdict=verdict,
+            status=resp.status_code,
+            response_time_ms=response_time_ms,
+            headers=headers,
+            cookies=cookies,
+            body_preview=body_preview,
+            tls_was_blocker=tls_was_blocker,
+            profile_used=profile,
+            full_body=resp.text
+        )
+    except Exception as e:
+        response_time_ms = int((time.perf_counter() - start_time) * 1000)
+        return TlsResult(
+            verdict=Verdict.ERROR,
+            status=None,
+            response_time_ms=response_time_ms,
+            headers={},
+            cookies=[],
+            body_preview="",
+            tls_was_blocker=False,
+            profile_used=profile,
+            error=str(e)
+        )

scraperecon-0.1.0/scraperecon/stages/vendor.py

@@ -0,0 +1,101 @@
+from ..types import VendorResult, Confidence, PlainResult, TlsResult, Verdict
+from ..signatures import load_signatures
+
+def run(plain_result: PlainResult, tls_result: TlsResult | None) -> VendorResult:
+    # Prefer Stage 2 result if it ran and was not skipped, else use Stage 1.
+    if tls_result and tls_result.verdict not in (Verdict.SKIPPED, Verdict.ERROR):
+        target_headers = tls_result.headers
+        target_cookies = tls_result.cookies
+        target_body = tls_result.body_preview
+        target_status = tls_result.status
+    else:
+        target_headers = plain_result.headers
+        target_cookies = plain_result.cookies
+        target_body = plain_result.body_preview
+        target_status = plain_result.status
+
+    sigs = load_signatures()
+    
+    all_scores = []
+    best_vendor = None
+    best_score = 0.0
+    best_signals = []
+    
+    target_body_lower = target_body.lower()
+    target_cookies_lower = [c.lower() for c in target_cookies]
+    
+    for vendor_data in sigs.get("vendors", []):
+        vendor_name = vendor_data["name"]
+        signals = vendor_data.get("signals", [])
+        
+        max_possible = sum(s.get("weight", 0) for s in signals)
+        if max_possible == 0:
+            continue
+            
+        score = 0.0
+        matched_signals = []
+        
+        for sig in signals:
+            stype = sig["type"]
+            weight = sig.get("weight", 0)
+            matched = False
+            match_str = ""
+            
+            if stype == "header_present":
+                key = sig["key"].lower()
+                if key in target_headers:
+                    matched = True
+                    match_str = f"{key} header"
+            elif stype == "header_value":
+                key = sig["key"].lower()
+                val = sig["value"].lower()
+                if key in target_headers and val in target_headers[key].lower():
+                    matched = True
+                    match_str = f"{val} in {key} header"
+            elif stype == "cookie_name":
+                val = sig["value"].lower()
+                if val in target_cookies_lower:
+                    matched = True
+                    match_str = f"{sig['value']} cookie"
+            elif stype == "body_contains":
+                val = sig["value"].lower()
+                if val in target_body_lower:
+                    matched = True
+                    match_str = f"{sig['value']} in body"
+            elif stype == "status_code":
+                if target_status == sig["value"]:
+                    matched = True
+                    match_str = f"status {sig['value']}"
+                    
+            if matched:
+                score += weight
+                matched_signals.append(match_str)
+                
+        normalized = score / max_possible
+        all_scores.append((vendor_name, normalized))
+        
+        if normalized > best_score:
+            best_score = normalized
+            best_vendor = vendor_name
+            best_signals = matched_signals
+
+    if best_score >= 0.8:
+        confidence = Confidence.HIGH
+    elif best_score >= 0.5:
+        confidence = Confidence.MEDIUM
+    elif best_score >= 0.3:
+        confidence = Confidence.LOW
+    else:
+        return VendorResult(
+            vendor=None,
+            confidence=None,
+            matched_signals=[],
+            all_scores=all_scores
+        )
+        
+    return VendorResult(
+        vendor=best_vendor,
+        confidence=confidence,
+        matched_signals=best_signals,
+        all_scores=all_scores
+    )

scraperecon-0.1.0/scraperecon/types.py

@@ -0,0 +1,82 @@
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Optional
+
+class Verdict(Enum):
+    OPEN = "Open"
+    BLOCKED = "Blocked"
+    CHALLENGED = "Challenged"
+    REDIRECTED = "Redirected"
+    UNCERTAIN = "Uncertain"
+    SKIPPED = "Skipped"
+    ERROR = "Error"
+
+class Confidence(Enum):
+    HIGH = "High"
+    MEDIUM = "Medium"
+    LOW = "Low"
+
+class BlockType(Enum):
+    HARD_BLOCK = "HardBlock"
+    RATE_LIMITED = "RateLimited"
+    SOFT_REDIRECT = "SoftRedirect"
+    SILENT = "Silent"
+
+@dataclass
+class PlainResult:
+    verdict: Verdict
+    status: Optional[int]
+    response_time_ms: int
+    headers: dict[str, str]
+    cookies: list[str]
+    body_preview: str
+    final_url: str
+    full_body: str = ""
+    error: Optional[str] = None
+
+@dataclass
+class TlsResult:
+    verdict: Verdict
+    status: Optional[int]
+    response_time_ms: int
+    headers: dict[str, str]
+    cookies: list[str]
+    body_preview: str
+    tls_was_blocker: bool
+    profile_used: str
+    full_body: str = ""
+    error: Optional[str] = None
+
+@dataclass
+class VendorResult:
+    vendor: Optional[str]
+    confidence: Optional[Confidence]
+    matched_signals: list[str]
+    all_scores: list[tuple[str, float]]
+
+@dataclass
+class RateLimitResult:
+    total_requests: int
+    successful: int
+    blocked: int
+    block_type: Optional[BlockType]
+    estimated_safe_rps: Optional[float]
+    retry_after_secs: Optional[int]
+    median_response_ms: int
+
+@dataclass
+class Recommendation:
+    use_tls_impersonation: bool
+    profile: Optional[str]
+    captcha_detected: bool
+    proxy_recommended: bool
+    notes: list[str] = field(default_factory=list)
+
+@dataclass
+class ReconReport:
+    target: str
+    plain: Optional[PlainResult]
+    tls: Optional[TlsResult]
+    vendor: Optional[VendorResult]
+    rate_limit: Optional[RateLimitResult]
+    recommendation: Recommendation

scraperecon-0.1.0/scraperecon/utils.py

@@ -0,0 +1,14 @@
+import json
+import importlib.resources
+from . import data
+
+def load_indicators() -> list[str]:
+    text = importlib.resources.read_text(data, "indicators.json")
+    return json.loads(text)
+
+def is_challenge_body(body: str) -> bool:
+    if not body:
+        return False
+    body_lower = body.lower()
+    indicators = load_indicators()
+    return any(indicator in body_lower for indicator in indicators)