hellhound-spider 14.1__py3-none-any.whl

hellhound_spider-14.1.data/data/share/man/man1/spider.1

@@ -0,0 +1,296 @@
+.TH SPIDER 1 "2026-11-01" "Hellhound Spider v14.1" "User Commands"
+.SH NAME
+spider \- SPA and non-SPA web crawler, endpoint discoverer, and attack-surface mapper
+.SH SYNOPSIS
+.B spider
+[\fIOPTIONS\fR] \fITARGET\fR
+.SH DESCRIPTION
+.B spider
+is a web application crawler and attack-surface mapper for offensive
+security assessments, bug bounty hunting, and CTF challenges. It
+performs a full-spectrum reconnaissance pass against a target, combining
+passive crawling, active probing, and multi-source intelligence gathering
+into a single run.
+.PP
+During a scan,
+.B spider
+follows links, parses forms and JavaScript, renders single-page
+applications via a headless browser, mines the Wayback Machine and
+certificate transparency logs, and probes for admin panels, sensitive
+files, GraphQL and OpenAPI endpoints, and directories from a
+user-supplied wordlist. Every discovered URL is recorded with its HTTP
+method, confidence level, and observed parameters.
+.PP
+After the forward crawl completes,
+.B spider
+also walks back up every discovered path to probe ancestor directories
+that were never directly linked to \(em for example, finding
+.I /js/
+open to directory listing after only ever seeing individual
+.I /js/*.min.js
+hits \(em the way a recursive directory brute-forcer would. This is on
+by default; see
+.BR \-k .
+.PP
+.B spider
+auto-detects whether its stdout is a terminal or a pipe. When piped
+(e.g.
+.BR "spider target.com | httpx" ),
+the banner, progress animation, and full colour-coded report are routed
+to stderr, and stdout carries only the structured report in the format
+requested with
+.BR \-f
+(plain newline-separated URLs by default). No extra flag is required
+for this; see the
+.B Piping
+section below.
+.PP
+The output includes a structured JSON report and a colour-coded
+terminal summary covering: response headers and missing security
+controls, TLS posture, detected technology stack, robots.txt and
+sitemap findings, Wayback-archived endpoints, HTML and JS comment leaks,
+a full parameter map per endpoint, authentication-walled paths, security
+findings (GraphQL introspection, CORS, source maps, exposed secrets),
+and intelligence classifications (admin panels, authentication
+endpoints, unauthenticated APIs, sensitive data sources, legacy
+interfaces, and file upload points). When a CTF flag format is supplied
+via
+.BR \-K ,
+matched flags are surfaced in a dedicated section at the top of the
+findings.
+.PP
+\fITARGET\fR is the base URL of the application to scan, e.g.
+.IR https://example.com .
+.SH OPTIONS
+.SS "Scan Options"
+.TP
+.BR \-d ", " \-\-depth " " \fIN\fR
+Maximum crawl depth (default: 4).
+.TP
+.BR \-W ", " \-\-delay " " \fIS\fR
+Fixed delay between requests per host, in seconds (default: 0, adaptive).
+.TP
+.BR \-c ", " \-\-concurrency " " \fIN\fR
+Number of concurrent crawl workers (default: 12).
+.TP
+.BR \-t ", " \-\-timeout " " \fIS\fR
+Per-request timeout in seconds (default: 15).
+.TP
+.BR \-v ", " \-\-verbose
+Show all discovery logs, including low-confidence and informational events.
+.SS "Authentication"
+.TP
+.BR \-C ", " \-\-cookie " " \fICOOKIE\fR
+Cookie string (e.g. \fB"session=42"\fR), dict-style string, or path to a cookie
+file. Used to scan authenticated areas of the application. Since the spider does
+not support automated form-login using username and password, you must manually
+log in via a browser and supply the session cookie here.
+.TP
+.BR \-a ", " \-\-auth " " \fIHEADER\fR
+Authorization header value, e.g. \fB"Bearer eyJ..."\fR.
+.TP
+.BR \-u ", " \-\-basic\-auth " " \fIUSER:PASS\fR
+HTTP Basic Access Authentication credentials, e.g. \fBadmin:password\fR. Note that
+this is for server-level basic authentication, not standard web login forms.
+.TP
+.BR \-X ", " \-\-header " " \fI"NAME: VALUE"\fR
+Custom HTTP header, repeatable. Pass custom headers formatted as \fB"Name: Value"\fR.
+Useful for program-required tester identification or custom tokens.
+.SS "Output"
+.TP
+.BR \-o ", " \-\-out " " \fIFILE\fR
+Write an additional copy of the report to \fIFILE\fR. A JSON report is
+always saved automatically regardless of this option.
+.TP
+.BR \-f ", " \-\-format " " "{json,jsonl,csv,burp,urls,nuclei}"
+Extra output format for the report written to \fB\-\-out\fR
+(default: \fBjson\fR).
+.SS "Feature Flags"
+.TP
+.BR \-P ", " \-\-no\-playwright
+Disable headless browser SPA scanning. Without this flag, \fBspider\fR
+attempts patchright, then playwright-stealth, then falls back to manual
+JavaScript analysis.
+.TP
+.BR \-p ", " \-\-probe
+Enable intelligent probing phase (extra method-discovery requests per
+endpoint). Off by default.
+.TP
+.BR \-m ", " \-\-admin\-probe
+Probe common admin/management panel paths. Off by default.
+.TP
+.BR \-e ", " \-\-sensitive\-probe
+Probe known sensitive file paths (.env, .git, backups, etc.). Off by default.
+.TP
+.BR \-y ", " \-\-wayback
+Query the Wayback Machine for archived URLs. Off by default.
+.TP
+.BR \-I ", " \-\-spa\-interact
+Enable SPA form filling and button clicking during headless browser
+scanning. Only use against targets you are authorized to test, as this
+performs active interactions with the application.
+.TP
+.BR \-J ", " \-\-deep\-spa
+After crawling, revisit every discovered page in a real browser to capture
+background XHR/fetch API calls (the "Network tab" view). Best for smaller targets
+with few sub-applications or lab instances — captures dynamic API endpoints
+that only fire when page JS executes.
+.TP
+.BR \-R ", " \-\-no\-cors
+Disable CORS misconfiguration checks.
+.TP
+.BR \-G ", " \-\-no\-graphql
+Disable GraphQL introspection probing.
+.TP
+.BR \-O ", " \-\-no\-openapi
+Disable OpenAPI / Swagger discovery.
+.TP
+.BR \-x ", " \-\-extract
+Enable passive data extraction of emails, IP addresses, and cloud storage
+bucket references found in crawled content.
+.TP
+.BR \-s ", " \-\-screenshot " " [\fIPRESET\fR]
+Capture screenshots of key endpoints. \fIPRESET\fR may be one of:
+\fBall\fR, \fBstandard\fR, \fBblocked\fR, \fBerrors\fR, \fBapi\fR,
+\fBadmin\fR, or a regular expression matching endpoint URLs. Defaults to
+\fBstandard\fR if no preset is given.
+.TP
+.BR \-F ", " \-\-no\-filter
+Disable the noise-path filter, including VCS browser UI paths, CDN asset
+paths, and socket.io endpoints in the output.
+.TP
+.BR \-N ", " \-\-no\-crawl
+Skip BFS link crawling entirely. Only the recon and probe modules run:
+robots.txt and sitemap parsing, admin panel probing, sensitive file
+probing, wordlist-based discovery (\fB\-\-wordlist\fR), subdomain
+enumeration (\fB\-\-subdomains\fR), and Wayback Machine history mining.
+Useful for fast, targeted reconnaissance without traversing the site's
+link graph.
+.TP
+.BR \-k ", " \-\-no\-backtrack
+Disable parent-directory backtracking. On by default: after the crawl,
+.B spider
+probes every ancestor directory of every discovered path (e.g. finding
+.I /js/
+open after only ever seeing individual
+.IR /js/*.min.js " hits)," and crawls whatever that turns up \(em
+listed directories, in particular, often expose files the forward
+crawl never had a link to.
+.SS "Piping"
+.TP
+.BR \-Q ", " \-\-pipe
+Force machine-pipe mode: the banner, progress animation, and full
+colour-coded report are routed to stderr; stdout carries only the
+structured report, in the format chosen with
+.BR \-f .
+This is auto-enabled whenever stdout is not a terminal, so
+.B "spider target.com | httpx"
+or
+.B "spider target.com | nuclei -t cves/"
+already work with no flag at all \(em pass
+.B \-Q
+explicitly to force clean stdout output even when stdout
+.I is
+a terminal. With multiple targets read from stdin and the default JSON
+format, each target's report is written as one compact line (JSONL).
+When piping, \fB\-f\fR defaults to \fBurls\fR instead of \fBjson\fR,
+since that's what most downstream tools expect.
+.SS "Scope"
+.TP
+.BR \-\-subdomains
+Enable subdomain enumeration via certificate transparency logs.
+.TP
+.BR \-S ", " \-\-follow\-subdomains
+Crawl discovered subdomains within the base domain.
+.TP
+.BR \-r ", " \-\-follow\-redirects
+Follow cross-host redirects and add the destination host to scope.
+.TP
+.BR \-A ", " \-\-scope " " \fIHOSTS\fR
+Comma-separated list of additional hosts to include in scope, e.g.
+\fBapi.target.com,cdn.target.com\fR.
+.TP
+.BR \-w ", " \-\-wordlist " " \fIFILE\fR
+Path to a newline-separated wordlist of directory and file names. Each
+entry is requested against the target; responses are filtered through the
+same soft-404 and canary-fingerprint logic used for sensitive-file
+detection, so only meaningful hits are reported. Combine with
+\fB\-\-no\-crawl\fR for a discovery-only pass.
+.SS "CTF"
+.TP
+.BR \-\-ctf
+Enable CTF Mode. Automatically sets target flag matching templates to common defaults (e.g. flag{}, FLAG{}, etc. if not explicitly set via \fB\-K\fR), and enables active probing (\fB\-p\fR), sensitive files scanning (\fB\-e\fR), admin panel probing (\fB\-m\fR), GraphQL/OpenAPI schema discovery, and CORS audits with a higher default concurrency of 20.
+.TP
+.BR \-K ", " \-\-ctf\-flag " " \fITEMPLATE\fR
+Flag format template to scan for across all crawled content (HTML, JS,
+CSS, JSON responses, error pages, and comments). Use \fB{}\fR as the
+placeholder for the flag body. Multiple formats may be given
+comma-separated, e.g. \fBflag{},ctf{}\fR.
+.SS "Utilities"
+.TP
+.BR \-D ", " \-\-diff " " \fIOLD_REPORT\fR
+Diff the results of this scan against a previous JSON report, highlighting
+new, removed, and changed endpoints.
+.TP
+.BR \-H ", " \-\-har " " \fIHAR_FILE\fR
+Import a browser HAR session file to seed the scan with authenticated or
+otherwise hard-to-reach requests captured during manual browsing.
+.TP
+.BR \-U ", " \-\-upgrade
+Upgrade \fBspider\fR to the latest version.
+.SS "General"
+.TP
+.BR \-h ", " \-\-help
+Show a summary of command-line options and exit.
+.SH OUTPUT
+.B spider
+always writes a JSON report to the working directory, named after the
+target host and scan timestamp. The on-screen summary includes:
+.PP
+.RS
+.IP \(bu 3
+response headers and missing security headers
+.IP \(bu 3
+TLS/certificate findings
+.IP \(bu 3
+detected technology stack
+.IP \(bu 3
+robots.txt disallowed paths and discovered endpoints
+.IP \(bu 3
+a parameter map of query, form, and runtime parameters per endpoint
+.IP \(bu 3
+authentication-walled endpoints
+.IP \(bu 3
+intelligence classifications: admin panels, IDOR candidates, file upload
+endpoints, authentication endpoints, unauthenticated APIs, sensitive data
+sources, and legacy/deprecated endpoints
+.RE
+.SH EXAMPLES
+.TP
+Scan a target with default settings:
+.B spider http://target.example
+.TP
+Authenticated scan using a session cookie:
+.B spider \-C "PHPSESSID=abc123" http://target.example
+.TP
+Recon-only pass with a custom wordlist, no link crawling:
+.B spider \-N \-\-wordlist ~/wordlists/common.txt http://target.example
+.TP
+Full reconnaissance including subdomain enumeration, no crawl:
+.B spider \-N \-\-subdomains \-\-wordlist ~/wordlists/common.txt http://target.example
+.TP
+SPA scan with headless browser interaction enabled:
+.B spider \-I https://app.example
+.TP
+Import a HAR capture and diff against a previous report:
+.B spider \-H session.har \-D previous_report.json http://target.example
+.TP
+Pipe discovered URLs straight into another tool, no extra flag needed:
+.B "spider http://target.example | httpx"
+.SH FILES
+.TP
+.I spider_<host>_<timestamp>.json
+Auto-saved JSON report for each scan.
+.SH AUTHOR
+.B spider
+was written by L4ZZ3RJ0D and is part of the Hellhound Pentest framework (https://github.com/project-hellhound-org).

hellhound_spider-14.1.dist-info/METADATA

@@ -0,0 +1,201 @@
+Metadata-Version: 2.4
+Name: hellhound-spider
+Version: 14.1
+Summary: Autonomous web crawler and recon engine for security testing — SPA + Non-SPA crawling, header analysis, subdomain enumeration, and intelligence classification
+Author-email: Sree Danush S <lazzer@gmail.com>
+License-Expression: GPL-3.0-or-later
+Project-URL: Homepage, https://github.com/project-hellhound-org/hellhound-spider
+Project-URL: Changelog, https://github.com/project-hellhound-org/hellhound-spider/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/project-hellhound-org/hellhound-spider/issues
+Keywords: security,pentest,crawler,web-crawler,spa,recon,reconnaissance,bug-bounty,ctf,attack-surface,endpoint-discovery
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Information Technology
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Security
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: lxml>=5.0.0
+Provides-Extra: spa
+Requires-Dist: playwright>=1.40.0; extra == "spa"
+Requires-Dist: playwright-stealth>=1.0.6; extra == "spa"
+Requires-Dist: patchright>=0.5.0; extra == "spa"
+Dynamic: license-file
+
+<p align="center">
+  <img src="Images/spider.jpg" alt="Hellhound Spider" width="600"/>
+</p>
+
+<h1 align="center">Hellhound Spider</h1>
+
+<p align="center">
+  Fully autonomous web crawler for security testing — maps endpoints, parameters, and security issues across traditional and SPA web applications. Drop a URL. Walk away.
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/python-3.10+-blue?style=flat-square&logo=python&logoColor=white"/>
+  <img src="https://img.shields.io/badge/version-14.1-red?style=flat-square"/>
+  <img src="https://img.shields.io/badge/platform-Linux%20%7C%20macOS%20%7C%20Windows-lightgrey?style=flat-square"/>
+  <img src="https://img.shields.io/badge/license-GPL--3.0-blue?style=flat-square"/>
+</p>
+
+---
+
+## What It Does
+
+Hellhound Spider crawls a web application and produces a complete map of every endpoint, parameter, and security surface it can reach. The output is a structured JSON report — sorted by confidence, with parameters grouped by source, ready to feed directly into attack agents or import into Burp Suite.
+
+It runs two crawl engines in parallel: async HTTP workers for speed, and headless Chromium for JavaScript-heavy SPAs. For SPAs it intercepts live XHR and fetch calls as the browser actually makes them — including POST body parameters and response IDs. Once the forward crawl is done, it also walks back up every discovered path to probe ancestor directories that were never directly linked to, the way a directory brute-forcer would. When the crawl finishes, it classifies every endpoint automatically so downstream agents start with context, not cold discovery.
+
+Output pipes straight into other tools — `spider https://target.com | httpx` just works, no extra flag needed.
+
+---
+
+## Installation
+
+### Linux / macOS
+
+```bash
+git clone https://github.com/project-hellhound-org/hellhound-spider.git
+cd hellhound-spider
+chmod +x install.sh
+./install.sh
+```
+
+The installer creates an isolated virtual environment (`.venv`) and a system-wide `spider` wrapper:
+
+```bash
+spider https://target.com
+```
+
+### Windows
+
+```bash
+git clone https://github.com/project-hellhound-org/hellhound-spider.git
+cd hellhound-spider
+pip install -e .                    # core install
+pip install -e ".[spa]"             # with Playwright SPA support
+playwright install chromium
+```
+
+### Uninstall
+
+```bash
+./uninstall.sh                   # Linux / macOS
+pip uninstall hellhound-spider   # Windows
+```
+
+---
+
+## Usage
+
+```bash
+spider <target> [options]
+```
+
+A handful of the flags you'll actually reach for day to day:
+
+```
+-d, --depth            Maximum crawl depth              (default: 4)
+-c, --concurrency       Concurrent async workers          (default: 12)
+-C, --cookie            Session cookie string or cookie file
+-a, --auth              Authorization header, e.g. "Bearer eyJ..."
+-x, --extract           Extract emails, IPs, cloud buckets, secrets
+-s, --screenshot        Capture screenshots (all/standard/api/admin/...)
+-f, --format            json / jsonl / csv / burp / urls / nuclei
+-o, --out               Output file path
+-v, --verbose           Show all discovery logs
+    --ctf                CTF mode — probes, CORS, flag scanning, high concurrency
+```
+
+That's the short list. Spider has a lot more surface than that — auth, scope control, modular probing, SPA interaction, HAR replay, subdomain enumeration, and more. Run `spider --help` or `man spider` for the full, always-current reference rather than relying on a table here that can drift out of sync with the tool.
+
+See [CHANGELOG.md](CHANGELOG.md) for version history.
+
+---
+
+## Examples
+
+```bash
+# Basic scan — drop a URL, spider does the rest
+spider https://target.com
+
+# Authenticated with a session cookie
+spider https://target.com -C "session=abc123; csrf=xyz"
+
+# Authenticated with Bearer token / HTTP Basic Auth
+spider https://target.com -a "Bearer eyJhbGci..."
+spider https://target.com -u "admin:password"
+
+# Pipe discovered URLs straight into another tool
+spider https://target.com | httpx
+spider https://target.com | nuclei -t cves/
+
+# Export for Burp Suite
+spider https://target.com -f burp -o burp.xml
+
+# Extraction and screenshots
+spider https://target.com -x -s all
+
+# Deeper crawl, all logs visible
+spider https://target.com -d 6 -v
+
+# Seed crawl with a HAR file (authenticated session replay)
+spider https://target.com --har session.har
+
+# Scan and extract specific CTF flags from all source files and comments
+spider https://target.com --ctf-flag "HELLCORP{},FLAG{}"
+```
+
+---
+
+## What Gets Found
+
+### Discovery Vectors
+
+HTML crawl, live SPA XHR interception, parent-directory backtracking (probes ancestor directories the forward crawl never linked to), Intelligent Robots Analysis (Disallow/Allow mapping + comment mining), sitemap XML (with index recursion), `.well-known` (OIDC/JWKS), JSON path chaining, SPA hash routes, lazy-load attributes, CSP header hints, OpenAPI/Swagger specs, GraphQL introspection, crt.sh certificate transparency, Wayback Machine CDX API, security.txt (RFC 9116), HAR file import, HTML comment mining, response header analysis, TLS/DNS intel, WAF fingerprinting, and JS SCA analysis.
+
+### Parameter Mining
+
+Form fields (with type metadata: hidden, file, required), JS fetch/axios body keys, URL query strings, OpenAPI spec fields, POST body params from live browser requests, structural normalization (clustering dynamic segments).
+
+---
+
+## Output Formats
+
+- **JSON** — Full-fidelity report with all classification metadata.
+- **Burp** — XML format for direct import into Burp Suite.
+- **CSV** — Spreadsheet-ready endpoint and parameter list.
+- **JSONL** — One endpoint per line for streaming pipelines.
+- **URLs** — Raw newline-separated list of discovered URLs (the default when piped to another tool).
+- **Nuclei** — Target list formatted for direct piping into Nuclei.
+
+---
+
+## Requirements
+
+- Python 3.10+
+- `aiohttp`, `beautifulsoup4`, `lxml`
+- Playwright + Chromium *(optional, for SPA targets)*
+- Patchright *(optional, automatic bot-bypass fallback when WAF blocks Playwright)*
+- WhatWeb *(optional, automatic system technology fingerprinter; installed by install.sh)*
+
+---
+
+For authorized security testing only. This software is licensed under the **GNU General Public License v3 (GPLv3)**.
+
+---
+
+## Author
+
+<a href="https://l4zz3rj0d.github.io">
+  <img src="https://img.shields.io/badge/Founder-L4ZZ3RJ0D-c0392b?style=for-the-badge" alt="L4ZZ3RJ0D"/>
+</a>

hellhound_spider-14.1.dist-info/RECORD

@@ -0,0 +1,8 @@
+spider.py,sha256=8e4KWD_Ms4Zd-3X2IA29q0bn4ZQn_Do_h3YAryendzE,464752
+hellhound_spider-14.1.data/data/share/man/man1/spider.1,sha256=guKy61YSbAkZxiGhgiQZvCYbF-zor4PS9o2RzmevzWE,11441
+hellhound_spider-14.1.dist-info/licenses/LICENSE,sha256=OXLcl0T2SZ8Pmy2_dmlvKuetivmyPd5m1q-Gyd-zaYY,35149
+hellhound_spider-14.1.dist-info/METADATA,sha256=WpPdRlspYwD89MKJDeLUjYh-xJ8mkC5mhip8TZ2Wj3c,8034
+hellhound_spider-14.1.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+hellhound_spider-14.1.dist-info/entry_points.txt,sha256=QaQLy_y3v9m2wbbGcpGe_lq3gQrK0EKwyt3dIsH8OWI,39
+hellhound_spider-14.1.dist-info/top_level.txt,sha256=-omgKN_8Zv_sz8Nh9RKvMeKEgG6ssFAFraIRpgIn8zA,7
+hellhound_spider-14.1.dist-info/RECORD,,

hellhound_spider-14.1.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

hellhound_spider-14.1.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+spider = spider:main