waymore 6.0__tar.gz → 6.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {waymore-6.0 → waymore-6.2}/PKG-INFO +12 -3
- waymore-6.0/waymore.egg-info/PKG-INFO → waymore-6.2/README.md +11 -17
- waymore-6.2/waymore/__init__.py +1 -0
- {waymore-6.0 → waymore-6.2}/waymore/waymore.py +65 -41
- waymore-6.0/README.md → waymore-6.2/waymore.egg-info/PKG-INFO +26 -2
- waymore-6.0/waymore/__init__.py +0 -1
- {waymore-6.0 → waymore-6.2}/LICENSE +0 -0
- {waymore-6.0 → waymore-6.2}/setup.cfg +0 -0
- {waymore-6.0 → waymore-6.2}/setup.py +0 -0
- {waymore-6.0 → waymore-6.2}/waymore.egg-info/SOURCES.txt +0 -0
- {waymore-6.0 → waymore-6.2}/waymore.egg-info/dependency_links.txt +0 -0
- {waymore-6.0 → waymore-6.2}/waymore.egg-info/entry_points.txt +0 -0
- {waymore-6.0 → waymore-6.2}/waymore.egg-info/requires.txt +0 -0
- {waymore-6.0 → waymore-6.2}/waymore.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: waymore
|
|
3
|
-
Version: 6.
|
|
3
|
+
Version: 6.2
|
|
4
4
|
Summary: Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan & VirusTotal!
|
|
5
5
|
Home-page: https://github.com/xnl-h4ck3r/waymore
|
|
6
6
|
Author: @xnl-h4ck3r
|
|
@@ -15,7 +15,7 @@ Requires-Dist: tldextract
|
|
|
15
15
|
|
|
16
16
|
<center><img src="https://github.com/xnl-h4ck3r/waymore/blob/main/waymore/images/title.png"></center>
|
|
17
17
|
|
|
18
|
-
## About - v6.
|
|
18
|
+
## About - v6.2
|
|
19
19
|
|
|
20
20
|
The idea behind **waymore** is to find even more links from the Wayback Machine than other existing tools.
|
|
21
21
|
|
|
@@ -109,6 +109,7 @@ pipx install git+https://github.com/xnl-h4ck3r/waymore.git
|
|
|
109
109
|
| -lr | --limit-requests | Limit the number of requests that will be made when getting links from a source (this doesn\'t apply to Common Crawl). Some targets can return a huge amount of requests needed that are just not feasible to get, so this can be used to manage that situation. This defaults to 0 (Zero) which means there is no limit. |
|
|
110
110
|
| -ow | --output-overwrite | If the URL output file (default `waymore.txt`, or specified by `-oU`) already exists, it will be overwritten instead of being appended to. |
|
|
111
111
|
| -nlf | --new-links-file | If this argument is passed, a `waymore.new` file (or if `-oU` is used it will be the name of that file suffixed with `.new`) will also be written, and will contain links for the latest run. This can be used for continuous monitoring of a target (only for `mode U`, not `mode R`). |
|
|
112
|
+
| | --stream | Output URLs to STDOUT as soon as they are found (duplicates will be shown). Only works with `-mode U`. All other output is suppressed, so use `-v` to see any errors. Use `-oU` to explicitly save results to file (wil be deduplicated). |
|
|
112
113
|
| -c | --config | Path to the YML config file. If not passed, it looks for file `config.yml` in the default directory, typically `~/.config/waymore`. |
|
|
113
114
|
| -wrlr | --wayback-rate-limit-retry | The number of minutes the user wants to wait for a rate limit pause on Wayback Machine (archive.org) instead of stopping with a `429` error (default: 3). |
|
|
114
115
|
| -urlr | --urlscan-rate-limit-retry | The number of minutes the user wants to wait for a rate limit pause on URLScan.io instead of stopping with a `429` error (default: 1). |
|
|
@@ -164,7 +165,7 @@ The `config.yml` file (typically in `~/.config/waymore/`) have values that can b
|
|
|
164
165
|
- `FILTER_KEYWORDS` - Only links and responses will be returned that contain the specified keywords if the `-ko`/`--keywords-only` argument is passed (without providing an explicit value on the command line), e.g. `admin,portal`
|
|
165
166
|
- `URLSCAN_API_KEY` - You can sign up to [urlscan.io](https://urlscan.io/user/signup) to get a **FREE** API key (there are also paid subscriptions available). It is recommended you get a key and put it into the config file so that you can get more back (and quicker) from their API. NOTE: You will get rate limited unless you have a full paid subscription.
|
|
166
167
|
- `CONTINUE_RESPONSES_IF_PIPED` - If retrieving archive responses doesn't complete, you will be prompted next time whether you want to continue with the previous run. However, if `stdout` is piped to another process it is assumed you don't want to have an interactive prompt. A value of `True` (default) will determine assure the previous run will be continued. if you want a fresh run every time then set to `False`.
|
|
167
|
-
- `WEBHOOK_DISCORD` - If the `--notify-discord` argument is passed, `
|
|
168
|
+
- `WEBHOOK_DISCORD` - If the `--notify-discord` argument is passed, `waymore` will send a notification to this Discord wehook when a successful XSS is found.
|
|
168
169
|
- `DEFAULT_OUTPUT_DIR` - This is the default location of any output files written if the `-oU` and `-oR` arguments are not used. If the value of this key is blank, then it will default to the location of the `config.yml` file.
|
|
169
170
|
- `INTELX_API_KEY` - You can sign up to [intelx.io here](https://intelx.io/product). It requires a paid API key to do the `/phonebook/search` through their API (as of 2024-09-01, the Phonebook service has been restricted to paid users due to constant abuse by spam accounts).
|
|
170
171
|
|
|
@@ -278,6 +279,14 @@ xnLinkFinder -i ~/Tools/waymore/results/redbull.com -sp https://www.redbull.com
|
|
|
278
279
|
|
|
279
280
|
Or run other tools such as [trufflehog](https://github.com/trufflesecurity/trufflehog) or [gf](https://github.com/tomnomnom/gf) over the directory of responses to find even more from the archived responses!
|
|
280
281
|
|
|
282
|
+
## In Depth Instructions
|
|
283
|
+
|
|
284
|
+
Below is an in-depth talk I did for [Jason Haddix's discord channel](https://discord.gg/jhaddix) back in March 2024 to cover **EVERYTHING** you need to know about `waymore`.
|
|
285
|
+
|
|
286
|
+
**NOTE: This video is from March 2024, so any features added after this will not be featured and some features may have changed. Please double check the current instructions.**
|
|
287
|
+
|
|
288
|
+
[](https://www.youtube.com/watch?v=hMaYSi9ErnM)
|
|
289
|
+
|
|
281
290
|
## Issues
|
|
282
291
|
|
|
283
292
|
If you come across any problems at all, or have ideas for improvements, please feel free to raise an issue on Github. If there is a problem, it will be useful if you can provide the exact command you ran and a detailed description of the problem. If possible, run with `-v` to reproduce the problem and let me know about any error messages that are given.
|
|
@@ -1,21 +1,6 @@
|
|
|
1
|
-
Metadata-Version: 2.1
|
|
2
|
-
Name: waymore
|
|
3
|
-
Version: 6.0
|
|
4
|
-
Summary: Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan & VirusTotal!
|
|
5
|
-
Home-page: https://github.com/xnl-h4ck3r/waymore
|
|
6
|
-
Author: @xnl-h4ck3r
|
|
7
|
-
Description-Content-Type: text/markdown
|
|
8
|
-
License-File: LICENSE
|
|
9
|
-
Requires-Dist: requests
|
|
10
|
-
Requires-Dist: pyyaml
|
|
11
|
-
Requires-Dist: termcolor
|
|
12
|
-
Requires-Dist: psutil
|
|
13
|
-
Requires-Dist: urlparse3
|
|
14
|
-
Requires-Dist: tldextract
|
|
15
|
-
|
|
16
1
|
<center><img src="https://github.com/xnl-h4ck3r/waymore/blob/main/waymore/images/title.png"></center>
|
|
17
2
|
|
|
18
|
-
## About - v6.
|
|
3
|
+
## About - v6.2
|
|
19
4
|
|
|
20
5
|
The idea behind **waymore** is to find even more links from the Wayback Machine than other existing tools.
|
|
21
6
|
|
|
@@ -109,6 +94,7 @@ pipx install git+https://github.com/xnl-h4ck3r/waymore.git
|
|
|
109
94
|
| -lr | --limit-requests | Limit the number of requests that will be made when getting links from a source (this doesn\'t apply to Common Crawl). Some targets can return a huge amount of requests needed that are just not feasible to get, so this can be used to manage that situation. This defaults to 0 (Zero) which means there is no limit. |
|
|
110
95
|
| -ow | --output-overwrite | If the URL output file (default `waymore.txt`, or specified by `-oU`) already exists, it will be overwritten instead of being appended to. |
|
|
111
96
|
| -nlf | --new-links-file | If this argument is passed, a `waymore.new` file (or if `-oU` is used it will be the name of that file suffixed with `.new`) will also be written, and will contain links for the latest run. This can be used for continuous monitoring of a target (only for `mode U`, not `mode R`). |
|
|
97
|
+
| | --stream | Output URLs to STDOUT as soon as they are found (duplicates will be shown). Only works with `-mode U`. All other output is suppressed, so use `-v` to see any errors. Use `-oU` to explicitly save results to file (wil be deduplicated). |
|
|
112
98
|
| -c | --config | Path to the YML config file. If not passed, it looks for file `config.yml` in the default directory, typically `~/.config/waymore`. |
|
|
113
99
|
| -wrlr | --wayback-rate-limit-retry | The number of minutes the user wants to wait for a rate limit pause on Wayback Machine (archive.org) instead of stopping with a `429` error (default: 3). |
|
|
114
100
|
| -urlr | --urlscan-rate-limit-retry | The number of minutes the user wants to wait for a rate limit pause on URLScan.io instead of stopping with a `429` error (default: 1). |
|
|
@@ -164,7 +150,7 @@ The `config.yml` file (typically in `~/.config/waymore/`) have values that can b
|
|
|
164
150
|
- `FILTER_KEYWORDS` - Only links and responses will be returned that contain the specified keywords if the `-ko`/`--keywords-only` argument is passed (without providing an explicit value on the command line), e.g. `admin,portal`
|
|
165
151
|
- `URLSCAN_API_KEY` - You can sign up to [urlscan.io](https://urlscan.io/user/signup) to get a **FREE** API key (there are also paid subscriptions available). It is recommended you get a key and put it into the config file so that you can get more back (and quicker) from their API. NOTE: You will get rate limited unless you have a full paid subscription.
|
|
166
152
|
- `CONTINUE_RESPONSES_IF_PIPED` - If retrieving archive responses doesn't complete, you will be prompted next time whether you want to continue with the previous run. However, if `stdout` is piped to another process it is assumed you don't want to have an interactive prompt. A value of `True` (default) will determine assure the previous run will be continued. if you want a fresh run every time then set to `False`.
|
|
167
|
-
- `WEBHOOK_DISCORD` - If the `--notify-discord` argument is passed, `
|
|
153
|
+
- `WEBHOOK_DISCORD` - If the `--notify-discord` argument is passed, `waymore` will send a notification to this Discord wehook when a successful XSS is found.
|
|
168
154
|
- `DEFAULT_OUTPUT_DIR` - This is the default location of any output files written if the `-oU` and `-oR` arguments are not used. If the value of this key is blank, then it will default to the location of the `config.yml` file.
|
|
169
155
|
- `INTELX_API_KEY` - You can sign up to [intelx.io here](https://intelx.io/product). It requires a paid API key to do the `/phonebook/search` through their API (as of 2024-09-01, the Phonebook service has been restricted to paid users due to constant abuse by spam accounts).
|
|
170
156
|
|
|
@@ -278,6 +264,14 @@ xnLinkFinder -i ~/Tools/waymore/results/redbull.com -sp https://www.redbull.com
|
|
|
278
264
|
|
|
279
265
|
Or run other tools such as [trufflehog](https://github.com/trufflesecurity/trufflehog) or [gf](https://github.com/tomnomnom/gf) over the directory of responses to find even more from the archived responses!
|
|
280
266
|
|
|
267
|
+
## In Depth Instructions
|
|
268
|
+
|
|
269
|
+
Below is an in-depth talk I did for [Jason Haddix's discord channel](https://discord.gg/jhaddix) back in March 2024 to cover **EVERYTHING** you need to know about `waymore`.
|
|
270
|
+
|
|
271
|
+
**NOTE: This video is from March 2024, so any features added after this will not be featured and some features may have changed. Please double check the current instructions.**
|
|
272
|
+
|
|
273
|
+
[](https://www.youtube.com/watch?v=hMaYSi9ErnM)
|
|
274
|
+
|
|
281
275
|
## Issues
|
|
282
276
|
|
|
283
277
|
If you come across any problems at all, or have ideas for improvements, please feel free to raise an issue on Github. If there is a problem, it will be useful if you can provide the exact command you ran and a detailed description of the problem. If possible, run with `-v` to reproduce the problem and let me know about any error messages that are given.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
__version__="6.2"
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
#!/usr/bin/env python
|
|
2
2
|
# Python 3
|
|
3
|
-
# waymore - by @Xnl-h4ck3r: Find way more from the Wayback Machine (also get links from Common Crawl, AlienVault OTX, URLScan and
|
|
3
|
+
# waymore - by @Xnl-h4ck3r: Find way more from the Wayback Machine (also get links from Common Crawl, AlienVault OTX, URLScan, VirusTotal and Intelligence X)
|
|
4
4
|
# Full help here: https://github.com/xnl-h4ck3r/waymore/blob/main/README.md
|
|
5
5
|
# Good luck and good hunting! If you really love the tool (or any others), or they helped you find an awesome bounty, consider BUYING ME A COFFEE! (https://ko-fi.com/xnlh4ck3r) ☕ (I could use the caffeine!)
|
|
6
6
|
|
|
@@ -205,7 +205,9 @@ def processStats():
|
|
|
205
205
|
def write(text='',pipe=False):
|
|
206
206
|
# Only send text to stdout if the tool isn't piped to pass output to something else,
|
|
207
207
|
# or if the tool has been piped and the pipe parameter is True
|
|
208
|
-
if
|
|
208
|
+
# AND if --stream is NOT active OR if it is active but we are explicitly piping (e.g. for URLs)
|
|
209
|
+
if (sys.stdout.isatty() or (not sys.stdout.isatty() and pipe)) and \
|
|
210
|
+
(not (args.stream and args.mode == 'U') or (args.stream and args.mode == 'U' and pipe)):
|
|
209
211
|
# If it has carriage return in the string, don't add a newline
|
|
210
212
|
if text.find('\r') > 0:
|
|
211
213
|
sys.stdout.write(text)
|
|
@@ -213,20 +215,22 @@ def write(text='',pipe=False):
|
|
|
213
215
|
sys.stdout.write(text+'\n')
|
|
214
216
|
|
|
215
217
|
def writerr(text='',pipe=False):
|
|
216
|
-
#
|
|
217
|
-
#
|
|
218
|
-
if
|
|
219
|
-
#
|
|
220
|
-
if text.find('\r') > 0:
|
|
221
|
-
sys.stdout.write(text)
|
|
222
|
-
else:
|
|
223
|
-
sys.stdout.write(text+'\n')
|
|
218
|
+
# If --stream is active and mode is 'U', and verbose is NOT true, suppress output.
|
|
219
|
+
# Otherwise, use the existing logic.
|
|
220
|
+
if args.stream and args.mode == 'U' and not args.verbose:
|
|
221
|
+
pass # Suppress output
|
|
224
222
|
else:
|
|
225
|
-
#
|
|
226
|
-
if
|
|
227
|
-
|
|
223
|
+
# Original logic: write to stdout if interactive, else stderr
|
|
224
|
+
if sys.stdout.isatty():
|
|
225
|
+
if text.find('\r') > 0:
|
|
226
|
+
sys.stdout.write(text)
|
|
227
|
+
else:
|
|
228
|
+
sys.stdout.write(text+'\n')
|
|
228
229
|
else:
|
|
229
|
-
|
|
230
|
+
if text.find('\r') > 0:
|
|
231
|
+
sys.stderr.write(text)
|
|
232
|
+
else:
|
|
233
|
+
sys.stderr.write(text+'\n')
|
|
230
234
|
|
|
231
235
|
def showVersion():
|
|
232
236
|
try:
|
|
@@ -710,27 +714,29 @@ def printProgressBar(
|
|
|
710
714
|
fill - Optional : bar fill character (Str)
|
|
711
715
|
printEnd - Optional : end character (e.g. "\r", "\r\n") (Str)
|
|
712
716
|
"""
|
|
713
|
-
|
|
714
|
-
|
|
715
|
-
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
write(colored(f"\r{prefix} |{bar}| {percent}% {suffix}\r", "green"))
|
|
722
|
-
else:
|
|
723
|
-
writerr(colored(f"\r{prefix} |{bar}| {percent}% {suffix}\r", "green"))
|
|
724
|
-
# Print New Line on Complete
|
|
725
|
-
if iteration == total:
|
|
717
|
+
# Only show progress bar if not streaming
|
|
718
|
+
if not (args.stream and args.mode == 'U'):
|
|
719
|
+
try:
|
|
720
|
+
percent = ("{0:." + str(decimals) + "f}").format(
|
|
721
|
+
100 * (iteration / float(total))
|
|
722
|
+
).rjust(5)
|
|
723
|
+
filledLength = int(length * iteration // total)
|
|
724
|
+
bar = fill * filledLength + "-" * (length - filledLength)
|
|
726
725
|
# If the program is not piped with something else, write to stdout, otherwise write to stderr
|
|
727
726
|
if sys.stdout.isatty():
|
|
728
|
-
write()
|
|
729
|
-
else:
|
|
730
|
-
writerr()
|
|
731
|
-
|
|
732
|
-
|
|
733
|
-
|
|
727
|
+
write(colored(f"\r{prefix} |{bar}| {percent}% {suffix}\r", "green"))
|
|
728
|
+
else:
|
|
729
|
+
writerr(colored(f"\r{prefix} |{bar}| {percent}% {suffix}\r", "green"))
|
|
730
|
+
# Print New Line on Complete
|
|
731
|
+
if iteration == total:
|
|
732
|
+
# If the program is not piped with something else, write to stdout, otherwise write to stderr
|
|
733
|
+
if sys.stdout.isatty():
|
|
734
|
+
write()
|
|
735
|
+
else:
|
|
736
|
+
writerr()
|
|
737
|
+
except Exception as e:
|
|
738
|
+
if verbose():
|
|
739
|
+
writerr(colored("ERROR printProgressBar: " + str(e), "red"))
|
|
734
740
|
|
|
735
741
|
def filehash(text):
|
|
736
742
|
"""
|
|
@@ -786,19 +792,25 @@ def linksFoundResponseAdd(link):
|
|
|
786
792
|
# Don't write it if the link does not contain the requested domain (this can sometimes happen)
|
|
787
793
|
if parsed_url.lower().find(checkInput.lower()) >= 0:
|
|
788
794
|
linksFound.add(link)
|
|
795
|
+
# If streaming is enabled and mode is 'U', print the link to stdout
|
|
796
|
+
if args.stream and args.mode == 'U':
|
|
797
|
+
write(link, pipe=True)
|
|
789
798
|
except Exception as e:
|
|
790
799
|
linksFound.add(link)
|
|
800
|
+
# If streaming is enabled and mode is 'U', print the link to stdout
|
|
801
|
+
if args.stream and args.mode == 'U':
|
|
802
|
+
write(link, pipe=True)
|
|
791
803
|
|
|
792
804
|
# Add a link to the linksFound collection
|
|
793
805
|
def linksFoundAdd(link):
|
|
794
806
|
global linksFound, argsInput, argsInputHostname
|
|
795
|
-
|
|
807
|
+
|
|
796
808
|
try:
|
|
797
809
|
if inputIsDomainANDPath:
|
|
798
810
|
checkInput = argsInput
|
|
799
811
|
else:
|
|
800
812
|
checkInput = argsInputHostname
|
|
801
|
-
|
|
813
|
+
|
|
802
814
|
# If the link specifies port 80 or 443, e.g. http://example.com:80, then remove the port
|
|
803
815
|
parsed = urlparse(link.strip())
|
|
804
816
|
if parsed.port in (80, 443):
|
|
@@ -806,12 +818,18 @@ def linksFoundAdd(link):
|
|
|
806
818
|
parsed_url = parsed._replace(netloc=new_netloc).geturl()
|
|
807
819
|
else:
|
|
808
820
|
parsed_url = link
|
|
809
|
-
|
|
821
|
+
|
|
810
822
|
# Don't write it if the link does not contain the requested domain (this can sometimes happen)
|
|
811
823
|
if parsed_url.find(checkInput) >= 0:
|
|
812
824
|
linksFound.add(link)
|
|
825
|
+
# If streaming is enabled and mode is 'U', print the link to stdout
|
|
826
|
+
if args.stream and args.mode == 'U':
|
|
827
|
+
write(link, pipe=True)
|
|
813
828
|
except Exception as e:
|
|
814
829
|
linksFound.add(link)
|
|
830
|
+
# If streaming is enabled and mode is 'U', print the link to stdout
|
|
831
|
+
if args.stream and args.mode == 'U':
|
|
832
|
+
write(link, pipe=True)
|
|
815
833
|
|
|
816
834
|
def processArchiveUrl(url):
|
|
817
835
|
"""
|
|
@@ -1062,7 +1080,7 @@ def processURLOutput():
|
|
|
1062
1080
|
else:
|
|
1063
1081
|
write(colored('\n-> Getting URLs (e.g. at 1 req/sec) could take more than '+str(days)+' days!!! Consider using arguments -lr, -ci, -from and -to wisely!','red'))
|
|
1064
1082
|
write('')
|
|
1065
|
-
|
|
1083
|
+
elif not (args.stream and args.mode == 'U' and args.output_urls == ''): # Only write to file if not streaming OR if streaming but -oU is provided
|
|
1066
1084
|
linkCount = len(linksFound)
|
|
1067
1085
|
write(getSPACER(colored('Links found for ' + subs + argsInput + ': ', 'cyan')+colored(str(linkCount) + ' 🤘','white'))+'\n')
|
|
1068
1086
|
|
|
@@ -1559,7 +1577,7 @@ def processURLScanUrl(url, httpCode, mimeType, urlscanID=''):
|
|
|
1559
1577
|
|
|
1560
1578
|
try:
|
|
1561
1579
|
# If the input has a / in it, then a URL was passed, so the link will only be added if the URL matches
|
|
1562
|
-
if
|
|
1580
|
+
if '/' in url:
|
|
1563
1581
|
if argsInput not in url:
|
|
1564
1582
|
addLink = False
|
|
1565
1583
|
|
|
@@ -3878,6 +3896,11 @@ def main():
|
|
|
3878
3896
|
action="store_true",
|
|
3879
3897
|
help="If this argument is passed, a .new file will also be written that will contain links for the latest run. This is only relevant for mode U.",
|
|
3880
3898
|
)
|
|
3899
|
+
parser.add_argument(
|
|
3900
|
+
'--stream',
|
|
3901
|
+
action='store_true',
|
|
3902
|
+
help='Output URLs to STDOUT as soon as they are found (duplicates will be shown). Only works with -mode U. All other output is suppressed, so use -v to see any errors. Use -oU to explicitly save results to file (wil be deduplicated).',
|
|
3903
|
+
)
|
|
3881
3904
|
parser.add_argument(
|
|
3882
3905
|
"-c",
|
|
3883
3906
|
"--config",
|
|
@@ -3924,7 +3947,7 @@ def main():
|
|
|
3924
3947
|
|
|
3925
3948
|
# If --version was passed, display version and exit
|
|
3926
3949
|
if args.version:
|
|
3927
|
-
|
|
3950
|
+
showVersion()
|
|
3928
3951
|
sys.exit()
|
|
3929
3952
|
|
|
3930
3953
|
# If -lcc wasn't passed then set to the default of 1 if -lcy is 0. This will make them work together
|
|
@@ -3975,7 +3998,8 @@ def main():
|
|
|
3975
3998
|
except:
|
|
3976
3999
|
pass
|
|
3977
4000
|
|
|
3978
|
-
|
|
4001
|
+
if not (args.stream and args.mode == 'U'):
|
|
4002
|
+
showBanner()
|
|
3979
4003
|
|
|
3980
4004
|
try:
|
|
3981
4005
|
|
|
@@ -4022,7 +4046,7 @@ def main():
|
|
|
4022
4046
|
# Get the config settings from the config.yml file
|
|
4023
4047
|
getConfig()
|
|
4024
4048
|
|
|
4025
|
-
if verbose():
|
|
4049
|
+
if verbose() and not (args.stream and args.mode == 'U'):
|
|
4026
4050
|
showOptions()
|
|
4027
4051
|
|
|
4028
4052
|
if args.check_only:
|
|
@@ -1,6 +1,21 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: waymore
|
|
3
|
+
Version: 6.2
|
|
4
|
+
Summary: Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan & VirusTotal!
|
|
5
|
+
Home-page: https://github.com/xnl-h4ck3r/waymore
|
|
6
|
+
Author: @xnl-h4ck3r
|
|
7
|
+
Description-Content-Type: text/markdown
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Requires-Dist: requests
|
|
10
|
+
Requires-Dist: pyyaml
|
|
11
|
+
Requires-Dist: termcolor
|
|
12
|
+
Requires-Dist: psutil
|
|
13
|
+
Requires-Dist: urlparse3
|
|
14
|
+
Requires-Dist: tldextract
|
|
15
|
+
|
|
1
16
|
<center><img src="https://github.com/xnl-h4ck3r/waymore/blob/main/waymore/images/title.png"></center>
|
|
2
17
|
|
|
3
|
-
## About - v6.
|
|
18
|
+
## About - v6.2
|
|
4
19
|
|
|
5
20
|
The idea behind **waymore** is to find even more links from the Wayback Machine than other existing tools.
|
|
6
21
|
|
|
@@ -94,6 +109,7 @@ pipx install git+https://github.com/xnl-h4ck3r/waymore.git
|
|
|
94
109
|
| -lr | --limit-requests | Limit the number of requests that will be made when getting links from a source (this doesn\'t apply to Common Crawl). Some targets can return a huge amount of requests needed that are just not feasible to get, so this can be used to manage that situation. This defaults to 0 (Zero) which means there is no limit. |
|
|
95
110
|
| -ow | --output-overwrite | If the URL output file (default `waymore.txt`, or specified by `-oU`) already exists, it will be overwritten instead of being appended to. |
|
|
96
111
|
| -nlf | --new-links-file | If this argument is passed, a `waymore.new` file (or if `-oU` is used it will be the name of that file suffixed with `.new`) will also be written, and will contain links for the latest run. This can be used for continuous monitoring of a target (only for `mode U`, not `mode R`). |
|
|
112
|
+
| | --stream | Output URLs to STDOUT as soon as they are found (duplicates will be shown). Only works with `-mode U`. All other output is suppressed, so use `-v` to see any errors. Use `-oU` to explicitly save results to file (wil be deduplicated). |
|
|
97
113
|
| -c | --config | Path to the YML config file. If not passed, it looks for file `config.yml` in the default directory, typically `~/.config/waymore`. |
|
|
98
114
|
| -wrlr | --wayback-rate-limit-retry | The number of minutes the user wants to wait for a rate limit pause on Wayback Machine (archive.org) instead of stopping with a `429` error (default: 3). |
|
|
99
115
|
| -urlr | --urlscan-rate-limit-retry | The number of minutes the user wants to wait for a rate limit pause on URLScan.io instead of stopping with a `429` error (default: 1). |
|
|
@@ -149,7 +165,7 @@ The `config.yml` file (typically in `~/.config/waymore/`) have values that can b
|
|
|
149
165
|
- `FILTER_KEYWORDS` - Only links and responses will be returned that contain the specified keywords if the `-ko`/`--keywords-only` argument is passed (without providing an explicit value on the command line), e.g. `admin,portal`
|
|
150
166
|
- `URLSCAN_API_KEY` - You can sign up to [urlscan.io](https://urlscan.io/user/signup) to get a **FREE** API key (there are also paid subscriptions available). It is recommended you get a key and put it into the config file so that you can get more back (and quicker) from their API. NOTE: You will get rate limited unless you have a full paid subscription.
|
|
151
167
|
- `CONTINUE_RESPONSES_IF_PIPED` - If retrieving archive responses doesn't complete, you will be prompted next time whether you want to continue with the previous run. However, if `stdout` is piped to another process it is assumed you don't want to have an interactive prompt. A value of `True` (default) will determine assure the previous run will be continued. if you want a fresh run every time then set to `False`.
|
|
152
|
-
- `WEBHOOK_DISCORD` - If the `--notify-discord` argument is passed, `
|
|
168
|
+
- `WEBHOOK_DISCORD` - If the `--notify-discord` argument is passed, `waymore` will send a notification to this Discord wehook when a successful XSS is found.
|
|
153
169
|
- `DEFAULT_OUTPUT_DIR` - This is the default location of any output files written if the `-oU` and `-oR` arguments are not used. If the value of this key is blank, then it will default to the location of the `config.yml` file.
|
|
154
170
|
- `INTELX_API_KEY` - You can sign up to [intelx.io here](https://intelx.io/product). It requires a paid API key to do the `/phonebook/search` through their API (as of 2024-09-01, the Phonebook service has been restricted to paid users due to constant abuse by spam accounts).
|
|
155
171
|
|
|
@@ -263,6 +279,14 @@ xnLinkFinder -i ~/Tools/waymore/results/redbull.com -sp https://www.redbull.com
|
|
|
263
279
|
|
|
264
280
|
Or run other tools such as [trufflehog](https://github.com/trufflesecurity/trufflehog) or [gf](https://github.com/tomnomnom/gf) over the directory of responses to find even more from the archived responses!
|
|
265
281
|
|
|
282
|
+
## In Depth Instructions
|
|
283
|
+
|
|
284
|
+
Below is an in-depth talk I did for [Jason Haddix's discord channel](https://discord.gg/jhaddix) back in March 2024 to cover **EVERYTHING** you need to know about `waymore`.
|
|
285
|
+
|
|
286
|
+
**NOTE: This video is from March 2024, so any features added after this will not be featured and some features may have changed. Please double check the current instructions.**
|
|
287
|
+
|
|
288
|
+
[](https://www.youtube.com/watch?v=hMaYSi9ErnM)
|
|
289
|
+
|
|
266
290
|
## Issues
|
|
267
291
|
|
|
268
292
|
If you come across any problems at all, or have ideas for improvements, please feel free to raise an issue on Github. If there is a problem, it will be useful if you can provide the exact command you ran and a detailed description of the problem. If possible, run with `-v` to reproduce the problem and let me know about any error messages that are given.
|
waymore-6.0/waymore/__init__.py
DELETED
|
@@ -1 +0,0 @@
|
|
|
1
|
-
__version__="6.0"
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|