livesub 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- livesub-0.1.0/PKG-INFO +93 -0
- livesub-0.1.0/README.md +80 -0
- livesub-0.1.0/pyproject.toml +25 -0
- livesub-0.1.0/setup.cfg +4 -0
- livesub-0.1.0/src/livesub/__init__.py +3 -0
- livesub-0.1.0/src/livesub/__main__.py +5 -0
- livesub-0.1.0/src/livesub/cli.py +442 -0
- livesub-0.1.0/src/livesub.egg-info/PKG-INFO +93 -0
- livesub-0.1.0/src/livesub.egg-info/SOURCES.txt +10 -0
- livesub-0.1.0/src/livesub.egg-info/dependency_links.txt +1 -0
- livesub-0.1.0/src/livesub.egg-info/entry_points.txt +2 -0
- livesub-0.1.0/src/livesub.egg-info/top_level.txt +1 -0
livesub-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: livesub
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Subdomain enumeration and live HTTP host reporting for authorized recon
|
|
5
|
+
Keywords: security,recon,subdomains,bug-bounty,pentesting
|
|
6
|
+
Classifier: Environment :: Console
|
|
7
|
+
Classifier: Intended Audience :: Information Technology
|
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
|
9
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
10
|
+
Classifier: Topic :: Security
|
|
11
|
+
Requires-Python: >=3.9
|
|
12
|
+
Description-Content-Type: text/markdown
|
|
13
|
+
|
|
14
|
+
# livesub
|
|
15
|
+
|
|
16
|
+
`livesub` is a Python command-line tool that runs ProjectDiscovery `subfinder`, probes discovered names with ProjectDiscovery `httpx`, and writes grouped text and JSON reports. It is intended for authorized pentesting and bug bounty work.
|
|
17
|
+
|
|
18
|
+
## Requirements
|
|
19
|
+
|
|
20
|
+
- Python 3.9 or newer
|
|
21
|
+
- ProjectDiscovery [subfinder](https://github.com/projectdiscovery/subfinder)
|
|
22
|
+
- ProjectDiscovery [httpx](https://github.com/projectdiscovery/httpx)
|
|
23
|
+
|
|
24
|
+
Install the scanners with Go:
|
|
25
|
+
|
|
26
|
+
```sh
|
|
27
|
+
go install github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
|
|
28
|
+
go install github.com/projectdiscovery/httpx/cmd/httpx@latest
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Make sure the Go `bin` directory is on your `PATH`. `livesub` has no Python runtime dependencies. It checks that `httpx` exposes ProjectDiscovery's scanner flags before it starts; some systems also have an unrelated Python CLI named `httpx`. If needed, specify the scanner directly with `--httpx-bin /path/to/httpx` or `LIVESUB_HTTPX_BIN`.
|
|
32
|
+
|
|
33
|
+
## Install
|
|
34
|
+
|
|
35
|
+
Install the published package:
|
|
36
|
+
|
|
37
|
+
```sh
|
|
38
|
+
python -m pip install livesub
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Or install from a source checkout:
|
|
42
|
+
|
|
43
|
+
```sh
|
|
44
|
+
python -m pip install .
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
You can also run `./livesub` directly from a checkout without installing the Python package.
|
|
48
|
+
|
|
49
|
+
## Usage
|
|
50
|
+
|
|
51
|
+
```text
|
|
52
|
+
livesub [OPTIONS] <domain>
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Examples:
|
|
56
|
+
|
|
57
|
+
```sh
|
|
58
|
+
livesub example.com
|
|
59
|
+
livesub -o ./results example.com
|
|
60
|
+
livesub -o ./results -t 40 -timeout 8 example.com
|
|
61
|
+
livesub -silent example.com
|
|
62
|
+
livesub --httpx-bin "$HOME/go/bin/httpx" example.com
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Options:
|
|
66
|
+
|
|
67
|
+
| Option | Description |
|
|
68
|
+
| --- | --- |
|
|
69
|
+
| `-o <dir>` | Output directory. Defaults to `./reconchain_<domain>_<timestamp>`. |
|
|
70
|
+
| `-t <n>` | Thread count passed to both scanners. |
|
|
71
|
+
| `-timeout <seconds>` | Per-request timeout passed to httpx. Defaults to 10 seconds. |
|
|
72
|
+
| `-silent` | Suppress progress messages and print only the final summary. |
|
|
73
|
+
| `--subfinder-bin <path>` | Select a non-default subfinder executable. |
|
|
74
|
+
| `--httpx-bin <path>` | Select ProjectDiscovery httpx explicitly. |
|
|
75
|
+
| `-h`, `--help` | Show usage information. |
|
|
76
|
+
| `--version` | Show the livesub version. |
|
|
77
|
+
|
|
78
|
+
## Output
|
|
79
|
+
|
|
80
|
+
The output directory contains:
|
|
81
|
+
|
|
82
|
+
- `subdomains.txt`: raw subfinder results.
|
|
83
|
+
- `live_hosts.txt`: raw httpx JSONL results with status, title, and technology detection enabled.
|
|
84
|
+
- `summary.txt`: Markdown report grouped and sorted by HTTP status code.
|
|
85
|
+
- `summary.json`: machine-readable report with the same status groups and status-family counts.
|
|
86
|
+
|
|
87
|
+
Example final stdout summary:
|
|
88
|
+
|
|
89
|
+
```text
|
|
90
|
+
Summary: 42 subdomains found, 17 live | 2xx: 11, 3xx: 2, 4xx: 3, 5xx: 1
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Use this only for domains you own or have explicit permission to assess.
|
livesub-0.1.0/README.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# livesub
|
|
2
|
+
|
|
3
|
+
`livesub` is a Python command-line tool that runs ProjectDiscovery `subfinder`, probes discovered names with ProjectDiscovery `httpx`, and writes grouped text and JSON reports. It is intended for authorized pentesting and bug bounty work.
|
|
4
|
+
|
|
5
|
+
## Requirements
|
|
6
|
+
|
|
7
|
+
- Python 3.9 or newer
|
|
8
|
+
- ProjectDiscovery [subfinder](https://github.com/projectdiscovery/subfinder)
|
|
9
|
+
- ProjectDiscovery [httpx](https://github.com/projectdiscovery/httpx)
|
|
10
|
+
|
|
11
|
+
Install the scanners with Go:
|
|
12
|
+
|
|
13
|
+
```sh
|
|
14
|
+
go install github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
|
|
15
|
+
go install github.com/projectdiscovery/httpx/cmd/httpx@latest
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Make sure the Go `bin` directory is on your `PATH`. `livesub` has no Python runtime dependencies. It checks that `httpx` exposes ProjectDiscovery's scanner flags before it starts; some systems also have an unrelated Python CLI named `httpx`. If needed, specify the scanner directly with `--httpx-bin /path/to/httpx` or `LIVESUB_HTTPX_BIN`.
|
|
19
|
+
|
|
20
|
+
## Install
|
|
21
|
+
|
|
22
|
+
Install the published package:
|
|
23
|
+
|
|
24
|
+
```sh
|
|
25
|
+
python -m pip install livesub
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Or install from a source checkout:
|
|
29
|
+
|
|
30
|
+
```sh
|
|
31
|
+
python -m pip install .
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
You can also run `./livesub` directly from a checkout without installing the Python package.
|
|
35
|
+
|
|
36
|
+
## Usage
|
|
37
|
+
|
|
38
|
+
```text
|
|
39
|
+
livesub [OPTIONS] <domain>
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Examples:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
livesub example.com
|
|
46
|
+
livesub -o ./results example.com
|
|
47
|
+
livesub -o ./results -t 40 -timeout 8 example.com
|
|
48
|
+
livesub -silent example.com
|
|
49
|
+
livesub --httpx-bin "$HOME/go/bin/httpx" example.com
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Options:
|
|
53
|
+
|
|
54
|
+
| Option | Description |
|
|
55
|
+
| --- | --- |
|
|
56
|
+
| `-o <dir>` | Output directory. Defaults to `./reconchain_<domain>_<timestamp>`. |
|
|
57
|
+
| `-t <n>` | Thread count passed to both scanners. |
|
|
58
|
+
| `-timeout <seconds>` | Per-request timeout passed to httpx. Defaults to 10 seconds. |
|
|
59
|
+
| `-silent` | Suppress progress messages and print only the final summary. |
|
|
60
|
+
| `--subfinder-bin <path>` | Select a non-default subfinder executable. |
|
|
61
|
+
| `--httpx-bin <path>` | Select ProjectDiscovery httpx explicitly. |
|
|
62
|
+
| `-h`, `--help` | Show usage information. |
|
|
63
|
+
| `--version` | Show the livesub version. |
|
|
64
|
+
|
|
65
|
+
## Output
|
|
66
|
+
|
|
67
|
+
The output directory contains:
|
|
68
|
+
|
|
69
|
+
- `subdomains.txt`: raw subfinder results.
|
|
70
|
+
- `live_hosts.txt`: raw httpx JSONL results with status, title, and technology detection enabled.
|
|
71
|
+
- `summary.txt`: Markdown report grouped and sorted by HTTP status code.
|
|
72
|
+
- `summary.json`: machine-readable report with the same status groups and status-family counts.
|
|
73
|
+
|
|
74
|
+
Example final stdout summary:
|
|
75
|
+
|
|
76
|
+
```text
|
|
77
|
+
Summary: 42 subdomains found, 17 live | 2xx: 11, 3xx: 2, 4xx: 3, 5xx: 1
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Use this only for domains you own or have explicit permission to assess.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=68"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "livesub"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "Subdomain enumeration and live HTTP host reporting for authorized recon"
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.9"
|
|
11
|
+
keywords = ["security", "recon", "subdomains", "bug-bounty", "pentesting"]
|
|
12
|
+
classifiers = [
|
|
13
|
+
"Environment :: Console",
|
|
14
|
+
"Intended Audience :: Information Technology",
|
|
15
|
+
"Programming Language :: Python :: 3",
|
|
16
|
+
"Programming Language :: Python :: 3 :: Only",
|
|
17
|
+
"Topic :: Security",
|
|
18
|
+
]
|
|
19
|
+
dependencies = []
|
|
20
|
+
|
|
21
|
+
[project.scripts]
|
|
22
|
+
livesub = "livesub.cli:main"
|
|
23
|
+
|
|
24
|
+
[tool.setuptools.packages.find]
|
|
25
|
+
where = ["src"]
|
livesub-0.1.0/setup.cfg
ADDED
|
@@ -0,0 +1,442 @@
|
|
|
1
|
+
"""Command-line implementation for livesub."""
|
|
2
|
+
|
|
3
|
+
from __future__ import annotations
|
|
4
|
+
|
|
5
|
+
import argparse
|
|
6
|
+
import json
|
|
7
|
+
import os
|
|
8
|
+
import re
|
|
9
|
+
import shutil
|
|
10
|
+
import subprocess
|
|
11
|
+
import sys
|
|
12
|
+
from datetime import datetime
|
|
13
|
+
from pathlib import Path
|
|
14
|
+
from typing import Any, Optional, Sequence
|
|
15
|
+
|
|
16
|
+
from . import __version__
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
DOMAIN_RE = re.compile(
|
|
20
|
+
r"(?:[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?\.)+"
|
|
21
|
+
r"[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?\Z"
|
|
22
|
+
)
|
|
23
|
+
SUBFINDER_INSTALL = "go install github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest"
|
|
24
|
+
HTTPX_INSTALL = "go install github.com/projectdiscovery/httpx/cmd/httpx@latest"
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
class LivesubError(Exception):
|
|
28
|
+
"""An error that should be shown without a Python traceback."""
|
|
29
|
+
|
|
30
|
+
|
|
31
|
+
class CommandError(LivesubError):
|
|
32
|
+
"""A failed external command and any relevant stderr output."""
|
|
33
|
+
|
|
34
|
+
def __init__(self, message: str, stderr: str = "") -> None:
|
|
35
|
+
super().__init__(message)
|
|
36
|
+
self.stderr = stderr.strip()
|
|
37
|
+
|
|
38
|
+
|
|
39
|
+
def positive_int(value: str) -> int:
|
|
40
|
+
try:
|
|
41
|
+
number = int(value, 10)
|
|
42
|
+
except ValueError as exc:
|
|
43
|
+
raise argparse.ArgumentTypeError("must be a positive integer") from exc
|
|
44
|
+
if number < 1:
|
|
45
|
+
raise argparse.ArgumentTypeError("must be a positive integer")
|
|
46
|
+
return number
|
|
47
|
+
|
|
48
|
+
|
|
49
|
+
def make_parser() -> argparse.ArgumentParser:
|
|
50
|
+
parser = argparse.ArgumentParser(
|
|
51
|
+
prog="livesub",
|
|
52
|
+
description=(
|
|
53
|
+
"Enumerate subdomains with ProjectDiscovery subfinder, probe them "
|
|
54
|
+
"with ProjectDiscovery httpx, and write grouped reports."
|
|
55
|
+
),
|
|
56
|
+
epilog="Use only for domains you own or are explicitly authorized to test.",
|
|
57
|
+
)
|
|
58
|
+
parser.add_argument(
|
|
59
|
+
"-o", metavar="DIR", dest="outdir",
|
|
60
|
+
help="output directory (default: ./reconchain_<domain>_<timestamp>)",
|
|
61
|
+
)
|
|
62
|
+
parser.add_argument(
|
|
63
|
+
"-t", metavar="N", type=positive_int,
|
|
64
|
+
help="thread count passed to subfinder and httpx (tool defaults if omitted)",
|
|
65
|
+
)
|
|
66
|
+
parser.add_argument(
|
|
67
|
+
"-timeout", "--timeout", metavar="SECONDS", type=positive_int, default=10,
|
|
68
|
+
help="per-request httpx timeout in seconds (default: 10)",
|
|
69
|
+
)
|
|
70
|
+
parser.add_argument(
|
|
71
|
+
"-silent", action="store_true",
|
|
72
|
+
help="suppress progress messages and print only the final summary",
|
|
73
|
+
)
|
|
74
|
+
parser.add_argument(
|
|
75
|
+
"--subfinder-bin", default=os.environ.get("LIVESUB_SUBFINDER_BIN"),
|
|
76
|
+
metavar="PATH", help="path/name of the subfinder executable",
|
|
77
|
+
)
|
|
78
|
+
parser.add_argument(
|
|
79
|
+
"--httpx-bin", default=os.environ.get("LIVESUB_HTTPX_BIN"),
|
|
80
|
+
metavar="PATH", help="path/name of ProjectDiscovery httpx (resolves name conflicts)",
|
|
81
|
+
)
|
|
82
|
+
parser.add_argument(
|
|
83
|
+
"--version", action="version", version="%(prog)s " + __version__,
|
|
84
|
+
)
|
|
85
|
+
parser.add_argument("domain", help="target domain, such as example.com")
|
|
86
|
+
return parser
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
def progress(silent: bool, message: str) -> None:
|
|
90
|
+
if not silent:
|
|
91
|
+
print("[*] " + message, file=sys.stderr)
|
|
92
|
+
|
|
93
|
+
|
|
94
|
+
def _resolve_executable(name_or_path: str, command_name: str) -> Optional[str]:
|
|
95
|
+
"""Resolve an executable name or explicit path and ensure it can be run."""
|
|
96
|
+
expanded = os.path.expanduser(name_or_path)
|
|
97
|
+
if os.path.dirname(expanded):
|
|
98
|
+
candidate = os.path.abspath(expanded)
|
|
99
|
+
if os.path.isfile(candidate) and os.access(candidate, os.X_OK):
|
|
100
|
+
return candidate
|
|
101
|
+
return None
|
|
102
|
+
return shutil.which(expanded or command_name)
|
|
103
|
+
|
|
104
|
+
|
|
105
|
+
def require_subfinder(requested: Optional[str]) -> str:
|
|
106
|
+
executable = _resolve_executable(requested or "subfinder", "subfinder")
|
|
107
|
+
if executable:
|
|
108
|
+
return executable
|
|
109
|
+
raise LivesubError(
|
|
110
|
+
'Required tool "subfinder" is missing or not executable. Install it with:\n'
|
|
111
|
+
" " + SUBFINDER_INSTALL
|
|
112
|
+
)
|
|
113
|
+
|
|
114
|
+
|
|
115
|
+
def _httpx_candidates(requested: Optional[str]) -> list[str]:
|
|
116
|
+
if requested:
|
|
117
|
+
resolved = _resolve_executable(requested, "httpx")
|
|
118
|
+
return [resolved] if resolved else []
|
|
119
|
+
|
|
120
|
+
candidates: list[str] = []
|
|
121
|
+
seen: set[str] = set()
|
|
122
|
+
for directory in os.environ.get("PATH", "").split(os.pathsep):
|
|
123
|
+
candidate = Path(directory or os.curdir) / "httpx"
|
|
124
|
+
if candidate.is_file() and os.access(candidate, os.X_OK):
|
|
125
|
+
resolved = str(candidate.resolve())
|
|
126
|
+
if resolved not in seen:
|
|
127
|
+
seen.add(resolved)
|
|
128
|
+
candidates.append(resolved)
|
|
129
|
+
|
|
130
|
+
# Go's default install location is useful when it is not yet on PATH.
|
|
131
|
+
go_default = Path.home() / "go" / "bin" / "httpx"
|
|
132
|
+
if go_default.is_file() and os.access(go_default, os.X_OK):
|
|
133
|
+
resolved = str(go_default.resolve())
|
|
134
|
+
if resolved not in seen:
|
|
135
|
+
candidates.append(resolved)
|
|
136
|
+
return candidates
|
|
137
|
+
|
|
138
|
+
|
|
139
|
+
def _is_projectdiscovery_httpx(executable: str) -> bool:
|
|
140
|
+
"""Check for the distinctive PD flags before running any target probes."""
|
|
141
|
+
try:
|
|
142
|
+
result = subprocess.run(
|
|
143
|
+
[executable, "-h"],
|
|
144
|
+
stdout=subprocess.PIPE,
|
|
145
|
+
stderr=subprocess.STDOUT,
|
|
146
|
+
text=True,
|
|
147
|
+
errors="replace",
|
|
148
|
+
timeout=8,
|
|
149
|
+
check=False,
|
|
150
|
+
)
|
|
151
|
+
except (OSError, subprocess.TimeoutExpired):
|
|
152
|
+
return False
|
|
153
|
+
help_text = result.stdout or ""
|
|
154
|
+
return "-tech-detect" in help_text and "-status-code" in help_text
|
|
155
|
+
|
|
156
|
+
|
|
157
|
+
def require_httpx(requested: Optional[str]) -> str:
|
|
158
|
+
candidates = _httpx_candidates(requested)
|
|
159
|
+
for candidate in candidates:
|
|
160
|
+
if _is_projectdiscovery_httpx(candidate):
|
|
161
|
+
return candidate
|
|
162
|
+
|
|
163
|
+
if requested and not candidates:
|
|
164
|
+
detail = "The configured executable could not be found or is not executable."
|
|
165
|
+
elif candidates:
|
|
166
|
+
detail = (
|
|
167
|
+
'An executable named "httpx" was found, but it does not expose the '
|
|
168
|
+
"ProjectDiscovery -status-code and -tech-detect flags."
|
|
169
|
+
)
|
|
170
|
+
else:
|
|
171
|
+
detail = 'The "httpx" executable was not found on PATH.'
|
|
172
|
+
raise LivesubError(
|
|
173
|
+
"Required tool ProjectDiscovery httpx is unavailable. " + detail + "\n"
|
|
174
|
+
"Install it with:\n " + HTTPX_INSTALL + "\n"
|
|
175
|
+
"If it is installed elsewhere, pass --httpx-bin /path/to/httpx or set "
|
|
176
|
+
"LIVESUB_HTTPX_BIN."
|
|
177
|
+
)
|
|
178
|
+
|
|
179
|
+
|
|
180
|
+
def run_subfinder(
|
|
181
|
+
executable: str,
|
|
182
|
+
domain: str,
|
|
183
|
+
threads: Optional[int],
|
|
184
|
+
output_path: Path,
|
|
185
|
+
) -> None:
|
|
186
|
+
command = [executable, "-d", domain, "-silent"]
|
|
187
|
+
if threads is not None:
|
|
188
|
+
command.extend(["-t", str(threads)])
|
|
189
|
+
try:
|
|
190
|
+
with output_path.open("w", encoding="utf-8", newline="") as output:
|
|
191
|
+
result = subprocess.run(
|
|
192
|
+
command,
|
|
193
|
+
stdout=output,
|
|
194
|
+
stderr=subprocess.PIPE,
|
|
195
|
+
text=True,
|
|
196
|
+
errors="replace",
|
|
197
|
+
check=False,
|
|
198
|
+
)
|
|
199
|
+
except OSError as exc:
|
|
200
|
+
raise CommandError("Could not start subfinder: " + str(exc)) from exc
|
|
201
|
+
if result.returncode != 0:
|
|
202
|
+
raise CommandError(
|
|
203
|
+
"subfinder failed while enumerating " + domain + ".",
|
|
204
|
+
result.stderr,
|
|
205
|
+
)
|
|
206
|
+
|
|
207
|
+
|
|
208
|
+
def run_httpx(
|
|
209
|
+
executable: str,
|
|
210
|
+
threads: Optional[int],
|
|
211
|
+
timeout: int,
|
|
212
|
+
input_path: Path,
|
|
213
|
+
output_path: Path,
|
|
214
|
+
) -> None:
|
|
215
|
+
command = [
|
|
216
|
+
executable,
|
|
217
|
+
"-silent",
|
|
218
|
+
"-status-code",
|
|
219
|
+
"-title",
|
|
220
|
+
"-tech-detect",
|
|
221
|
+
"-json",
|
|
222
|
+
]
|
|
223
|
+
if threads is not None:
|
|
224
|
+
command.extend(["-t", str(threads)])
|
|
225
|
+
command.extend(["-timeout", str(timeout)])
|
|
226
|
+
try:
|
|
227
|
+
with input_path.open("r", encoding="utf-8") as source:
|
|
228
|
+
with output_path.open("w", encoding="utf-8", newline="") as output:
|
|
229
|
+
result = subprocess.run(
|
|
230
|
+
command,
|
|
231
|
+
stdin=source,
|
|
232
|
+
stdout=output,
|
|
233
|
+
stderr=subprocess.PIPE,
|
|
234
|
+
text=True,
|
|
235
|
+
errors="replace",
|
|
236
|
+
check=False,
|
|
237
|
+
)
|
|
238
|
+
except OSError as exc:
|
|
239
|
+
raise CommandError("Could not start ProjectDiscovery httpx: " + str(exc)) from exc
|
|
240
|
+
if result.returncode != 0:
|
|
241
|
+
raise CommandError("httpx failed while probing hosts.", result.stderr)
|
|
242
|
+
|
|
243
|
+
|
|
244
|
+
def count_nonempty_lines(path: Path) -> int:
|
|
245
|
+
with path.open("r", encoding="utf-8", errors="replace") as source:
|
|
246
|
+
return sum(1 for line in source if line.strip())
|
|
247
|
+
|
|
248
|
+
|
|
249
|
+
def _hostname(record: dict[str, Any]) -> str:
|
|
250
|
+
candidate = record.get("input") or record.get("host") or record.get("url") or ""
|
|
251
|
+
hostname = str(candidate).strip()
|
|
252
|
+
hostname = re.sub(r"^https?://", "", hostname, flags=re.IGNORECASE)
|
|
253
|
+
hostname = hostname.split("/", 1)[0]
|
|
254
|
+
hostname = re.sub(r":[0-9]+$", "", hostname)
|
|
255
|
+
return hostname
|
|
256
|
+
|
|
257
|
+
|
|
258
|
+
def load_results(path: Path) -> list[dict[str, Any]]:
|
|
259
|
+
results: list[dict[str, Any]] = []
|
|
260
|
+
with path.open("r", encoding="utf-8", errors="replace") as source:
|
|
261
|
+
for line_number, line in enumerate(source, 1):
|
|
262
|
+
if not line.strip():
|
|
263
|
+
continue
|
|
264
|
+
try:
|
|
265
|
+
record = json.loads(line)
|
|
266
|
+
except json.JSONDecodeError as exc:
|
|
267
|
+
raise LivesubError(
|
|
268
|
+
"Could not parse httpx JSONL at line " + str(line_number) + ": " + str(exc)
|
|
269
|
+
) from exc
|
|
270
|
+
if not isinstance(record, dict):
|
|
271
|
+
continue
|
|
272
|
+
try:
|
|
273
|
+
status_code = int(record.get("status_code") or 0)
|
|
274
|
+
except (TypeError, ValueError):
|
|
275
|
+
continue
|
|
276
|
+
if status_code < 1:
|
|
277
|
+
continue
|
|
278
|
+
technologies = record.get("tech") or []
|
|
279
|
+
if not isinstance(technologies, list):
|
|
280
|
+
technologies = [technologies]
|
|
281
|
+
results.append(
|
|
282
|
+
{
|
|
283
|
+
"hostname": _hostname(record),
|
|
284
|
+
"status_code": status_code,
|
|
285
|
+
"title": str(record.get("title") or ""),
|
|
286
|
+
"technologies": sorted(str(item) for item in technologies),
|
|
287
|
+
}
|
|
288
|
+
)
|
|
289
|
+
return sorted(results, key=lambda item: (item["status_code"], item["hostname"].casefold()))
|
|
290
|
+
|
|
291
|
+
|
|
292
|
+
def build_summary(domain: str, total_subdomains: int, results: list[dict[str, Any]]) -> dict[str, Any]:
|
|
293
|
+
families = {family: 0 for family in ("2xx", "3xx", "4xx", "5xx")}
|
|
294
|
+
grouped: dict[int, list[dict[str, Any]]] = {}
|
|
295
|
+
for result in results:
|
|
296
|
+
status_code = result["status_code"]
|
|
297
|
+
grouped.setdefault(status_code, []).append(result)
|
|
298
|
+
family = str(status_code // 100) + "xx"
|
|
299
|
+
if family in families:
|
|
300
|
+
families[family] += 1
|
|
301
|
+
return {
|
|
302
|
+
"domain": domain,
|
|
303
|
+
"generated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
|
|
304
|
+
"total_subdomains": total_subdomains,
|
|
305
|
+
"total_live": len(results),
|
|
306
|
+
"status_families": families,
|
|
307
|
+
"by_status": [
|
|
308
|
+
{"status_code": status_code, "hosts": grouped[status_code]}
|
|
309
|
+
for status_code in sorted(grouped)
|
|
310
|
+
],
|
|
311
|
+
}
|
|
312
|
+
|
|
313
|
+
|
|
314
|
+
def _markdown_cell(value: Any) -> str:
|
|
315
|
+
return str(value).replace("\\", "\\\\").replace("|", "\\|").replace("\r", " ").replace("\n", " ")
|
|
316
|
+
|
|
317
|
+
|
|
318
|
+
def render_text(summary: dict[str, Any]) -> str:
|
|
319
|
+
families = summary["status_families"]
|
|
320
|
+
lines = [
|
|
321
|
+
"# Recon summary: " + summary["domain"],
|
|
322
|
+
"",
|
|
323
|
+
"- Generated: " + summary["generated_at"],
|
|
324
|
+
"- Subdomains found: " + str(summary["total_subdomains"]),
|
|
325
|
+
"- Live hosts: " + str(summary["total_live"]),
|
|
326
|
+
"- Status families: " + ", ".join(f"{name}={families[name]}" for name in ("2xx", "3xx", "4xx", "5xx")),
|
|
327
|
+
"",
|
|
328
|
+
]
|
|
329
|
+
if not summary["by_status"]:
|
|
330
|
+
lines.append("_No live hosts found._")
|
|
331
|
+
return "\n".join(lines) + "\n"
|
|
332
|
+
|
|
333
|
+
for group in summary["by_status"]:
|
|
334
|
+
lines.extend(
|
|
335
|
+
[
|
|
336
|
+
"### HTTP " + str(group["status_code"]),
|
|
337
|
+
"",
|
|
338
|
+
"| Hostname | Status code | Title | Technologies |",
|
|
339
|
+
"| --- | ---: | --- | --- |",
|
|
340
|
+
]
|
|
341
|
+
)
|
|
342
|
+
for host in group["hosts"]:
|
|
343
|
+
lines.append(
|
|
344
|
+
"| "
|
|
345
|
+
+ " | ".join(
|
|
346
|
+
(
|
|
347
|
+
_markdown_cell(host["hostname"]),
|
|
348
|
+
str(host["status_code"]),
|
|
349
|
+
_markdown_cell(host["title"]),
|
|
350
|
+
_markdown_cell(", ".join(host["technologies"])),
|
|
351
|
+
)
|
|
352
|
+
)
|
|
353
|
+
+ " |"
|
|
354
|
+
)
|
|
355
|
+
lines.append("")
|
|
356
|
+
return "\n".join(lines) + "\n"
|
|
357
|
+
|
|
358
|
+
|
|
359
|
+
def write_reports(
|
|
360
|
+
domain: str,
|
|
361
|
+
total_subdomains: int,
|
|
362
|
+
live_hosts_path: Path,
|
|
363
|
+
summary_text_path: Path,
|
|
364
|
+
summary_json_path: Path,
|
|
365
|
+
) -> dict[str, Any]:
|
|
366
|
+
results = load_results(live_hosts_path)
|
|
367
|
+
summary = build_summary(domain, total_subdomains, results)
|
|
368
|
+
summary_json_path.write_text(
|
|
369
|
+
json.dumps(summary, indent=2, ensure_ascii=False) + "\n",
|
|
370
|
+
encoding="utf-8",
|
|
371
|
+
)
|
|
372
|
+
summary_text_path.write_text(render_text(summary), encoding="utf-8")
|
|
373
|
+
return summary
|
|
374
|
+
|
|
375
|
+
|
|
376
|
+
def _print_error(error: LivesubError, silent: bool) -> None:
|
|
377
|
+
print("Error: " + str(error), file=sys.stderr)
|
|
378
|
+
if isinstance(error, CommandError) and error.stderr and not silent:
|
|
379
|
+
print(error.stderr, file=sys.stderr, end="" if error.stderr.endswith("\n") else "\n")
|
|
380
|
+
|
|
381
|
+
|
|
382
|
+
def main(argv: Optional[Sequence[str]] = None) -> int:
|
|
383
|
+
parser = make_parser()
|
|
384
|
+
args = parser.parse_args(argv)
|
|
385
|
+
if not DOMAIN_RE.fullmatch(args.domain):
|
|
386
|
+
parser.error(f"{args.domain!r} does not look like a domain name")
|
|
387
|
+
|
|
388
|
+
try:
|
|
389
|
+
subfinder = require_subfinder(args.subfinder_bin)
|
|
390
|
+
httpx = require_httpx(args.httpx_bin)
|
|
391
|
+
except LivesubError as error:
|
|
392
|
+
_print_error(error, args.silent)
|
|
393
|
+
return 1
|
|
394
|
+
|
|
395
|
+
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
|
396
|
+
output_dir = Path(args.outdir or f"./reconchain_{args.domain}_{timestamp}")
|
|
397
|
+
try:
|
|
398
|
+
output_dir.mkdir(parents=True, exist_ok=True)
|
|
399
|
+
except OSError as exc:
|
|
400
|
+
print(f"Error: could not create output directory {output_dir}: {exc}", file=sys.stderr)
|
|
401
|
+
return 1
|
|
402
|
+
|
|
403
|
+
subdomains_path = output_dir / "subdomains.txt"
|
|
404
|
+
live_hosts_path = output_dir / "live_hosts.txt"
|
|
405
|
+
summary_text_path = output_dir / "summary.txt"
|
|
406
|
+
summary_json_path = output_dir / "summary.json"
|
|
407
|
+
|
|
408
|
+
progress(args.silent, f"Enumerating subdomains for {args.domain}...")
|
|
409
|
+
try:
|
|
410
|
+
run_subfinder(subfinder, args.domain, args.t, subdomains_path)
|
|
411
|
+
total_subdomains = count_nonempty_lines(subdomains_path)
|
|
412
|
+
live_hosts_path.write_text("", encoding="utf-8")
|
|
413
|
+
|
|
414
|
+
if total_subdomains:
|
|
415
|
+
progress(args.silent, f"Probing {total_subdomains} subdomains with ProjectDiscovery httpx...")
|
|
416
|
+
run_httpx(httpx, args.t, args.timeout, subdomains_path, live_hosts_path)
|
|
417
|
+
else:
|
|
418
|
+
progress(args.silent, "No subdomains found; skipping httpx.")
|
|
419
|
+
|
|
420
|
+
progress(args.silent, "Building grouped text and JSON reports...")
|
|
421
|
+
summary = write_reports(
|
|
422
|
+
args.domain,
|
|
423
|
+
total_subdomains,
|
|
424
|
+
live_hosts_path,
|
|
425
|
+
summary_text_path,
|
|
426
|
+
summary_json_path,
|
|
427
|
+
)
|
|
428
|
+
except (LivesubError, OSError) as error:
|
|
429
|
+
_print_error(error if isinstance(error, LivesubError) else LivesubError(str(error)), args.silent)
|
|
430
|
+
return 1
|
|
431
|
+
|
|
432
|
+
families = summary["status_families"]
|
|
433
|
+
print(
|
|
434
|
+
f"Summary: {summary['total_subdomains']} subdomains found, "
|
|
435
|
+
f"{summary['total_live']} live | 2xx: {families['2xx']}, "
|
|
436
|
+
f"3xx: {families['3xx']}, 4xx: {families['4xx']}, 5xx: {families['5xx']}"
|
|
437
|
+
)
|
|
438
|
+
return 0
|
|
439
|
+
|
|
440
|
+
|
|
441
|
+
if __name__ == "__main__":
|
|
442
|
+
raise SystemExit(main())
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: livesub
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Subdomain enumeration and live HTTP host reporting for authorized recon
|
|
5
|
+
Keywords: security,recon,subdomains,bug-bounty,pentesting
|
|
6
|
+
Classifier: Environment :: Console
|
|
7
|
+
Classifier: Intended Audience :: Information Technology
|
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
|
9
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
10
|
+
Classifier: Topic :: Security
|
|
11
|
+
Requires-Python: >=3.9
|
|
12
|
+
Description-Content-Type: text/markdown
|
|
13
|
+
|
|
14
|
+
# livesub
|
|
15
|
+
|
|
16
|
+
`livesub` is a Python command-line tool that runs ProjectDiscovery `subfinder`, probes discovered names with ProjectDiscovery `httpx`, and writes grouped text and JSON reports. It is intended for authorized pentesting and bug bounty work.
|
|
17
|
+
|
|
18
|
+
## Requirements
|
|
19
|
+
|
|
20
|
+
- Python 3.9 or newer
|
|
21
|
+
- ProjectDiscovery [subfinder](https://github.com/projectdiscovery/subfinder)
|
|
22
|
+
- ProjectDiscovery [httpx](https://github.com/projectdiscovery/httpx)
|
|
23
|
+
|
|
24
|
+
Install the scanners with Go:
|
|
25
|
+
|
|
26
|
+
```sh
|
|
27
|
+
go install github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
|
|
28
|
+
go install github.com/projectdiscovery/httpx/cmd/httpx@latest
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Make sure the Go `bin` directory is on your `PATH`. `livesub` has no Python runtime dependencies. It checks that `httpx` exposes ProjectDiscovery's scanner flags before it starts; some systems also have an unrelated Python CLI named `httpx`. If needed, specify the scanner directly with `--httpx-bin /path/to/httpx` or `LIVESUB_HTTPX_BIN`.
|
|
32
|
+
|
|
33
|
+
## Install
|
|
34
|
+
|
|
35
|
+
Install the published package:
|
|
36
|
+
|
|
37
|
+
```sh
|
|
38
|
+
python -m pip install livesub
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Or install from a source checkout:
|
|
42
|
+
|
|
43
|
+
```sh
|
|
44
|
+
python -m pip install .
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
You can also run `./livesub` directly from a checkout without installing the Python package.
|
|
48
|
+
|
|
49
|
+
## Usage
|
|
50
|
+
|
|
51
|
+
```text
|
|
52
|
+
livesub [OPTIONS] <domain>
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Examples:
|
|
56
|
+
|
|
57
|
+
```sh
|
|
58
|
+
livesub example.com
|
|
59
|
+
livesub -o ./results example.com
|
|
60
|
+
livesub -o ./results -t 40 -timeout 8 example.com
|
|
61
|
+
livesub -silent example.com
|
|
62
|
+
livesub --httpx-bin "$HOME/go/bin/httpx" example.com
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Options:
|
|
66
|
+
|
|
67
|
+
| Option | Description |
|
|
68
|
+
| --- | --- |
|
|
69
|
+
| `-o <dir>` | Output directory. Defaults to `./reconchain_<domain>_<timestamp>`. |
|
|
70
|
+
| `-t <n>` | Thread count passed to both scanners. |
|
|
71
|
+
| `-timeout <seconds>` | Per-request timeout passed to httpx. Defaults to 10 seconds. |
|
|
72
|
+
| `-silent` | Suppress progress messages and print only the final summary. |
|
|
73
|
+
| `--subfinder-bin <path>` | Select a non-default subfinder executable. |
|
|
74
|
+
| `--httpx-bin <path>` | Select ProjectDiscovery httpx explicitly. |
|
|
75
|
+
| `-h`, `--help` | Show usage information. |
|
|
76
|
+
| `--version` | Show the livesub version. |
|
|
77
|
+
|
|
78
|
+
## Output
|
|
79
|
+
|
|
80
|
+
The output directory contains:
|
|
81
|
+
|
|
82
|
+
- `subdomains.txt`: raw subfinder results.
|
|
83
|
+
- `live_hosts.txt`: raw httpx JSONL results with status, title, and technology detection enabled.
|
|
84
|
+
- `summary.txt`: Markdown report grouped and sorted by HTTP status code.
|
|
85
|
+
- `summary.json`: machine-readable report with the same status groups and status-family counts.
|
|
86
|
+
|
|
87
|
+
Example final stdout summary:
|
|
88
|
+
|
|
89
|
+
```text
|
|
90
|
+
Summary: 42 subdomains found, 17 live | 2xx: 11, 3xx: 2, 4xx: 3, 5xx: 1
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Use this only for domains you own or have explicit permission to assess.
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
README.md
|
|
2
|
+
pyproject.toml
|
|
3
|
+
src/livesub/__init__.py
|
|
4
|
+
src/livesub/__main__.py
|
|
5
|
+
src/livesub/cli.py
|
|
6
|
+
src/livesub.egg-info/PKG-INFO
|
|
7
|
+
src/livesub.egg-info/SOURCES.txt
|
|
8
|
+
src/livesub.egg-info/dependency_links.txt
|
|
9
|
+
src/livesub.egg-info/entry_points.txt
|
|
10
|
+
src/livesub.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
livesub
|