waymore 4.5__tar.gz → 7.6__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (19) hide show
  1. {waymore-4.5/waymore.egg-info → waymore-7.6}/PKG-INFO +53 -24
  2. {waymore-4.5 → waymore-7.6}/README.md +301 -278
  3. waymore-7.6/pyproject.toml +31 -0
  4. waymore-4.5/waymore.egg-info/requires.txt → waymore-7.6/requirements.txt +3 -1
  5. waymore-7.6/setup.py +80 -0
  6. waymore-7.6/tests/test_import.py +5 -0
  7. waymore-7.6/waymore/__init__.py +1 -0
  8. waymore-7.6/waymore/waymore.py +6839 -0
  9. {waymore-4.5 → waymore-7.6/waymore.egg-info}/PKG-INFO +53 -24
  10. {waymore-4.5 → waymore-7.6}/waymore.egg-info/SOURCES.txt +3 -0
  11. waymore-7.6/waymore.egg-info/requires.txt +8 -0
  12. waymore-4.5/setup.py +0 -48
  13. waymore-4.5/waymore/__init__.py +0 -1
  14. waymore-4.5/waymore/waymore.py +0 -3322
  15. {waymore-4.5 → waymore-7.6}/LICENSE +0 -0
  16. {waymore-4.5 → waymore-7.6}/setup.cfg +0 -0
  17. {waymore-4.5 → waymore-7.6}/waymore.egg-info/dependency_links.txt +0 -0
  18. {waymore-4.5 → waymore-7.6}/waymore.egg-info/entry_points.txt +0 -0
  19. {waymore-4.5 → waymore-7.6}/waymore.egg-info/top_level.txt +0 -0
{waymore-4.5/waymore.egg-info → waymore-7.6}/PKG-INFO RENAMED
@@ -1,29 +1,35 @@
1
- Metadata-Version: 2.1
1
+ Metadata-Version: 2.4
2
2
  Name: waymore
3
- Version: 4.5
4
- Summary: Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan & VirusTotal!
3
+ Version: 7.6
4
+ Summary: Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, VirusTotal & Intelligence X!
5
5
  Home-page: https://github.com/xnl-h4ck3r/waymore
6
- Author: @xnl-h4ck3r
6
+ Author: xnl-h4ck3r
7
+ License: MIT
8
+ Requires-Python: >=3.9
7
9
  Description-Content-Type: text/markdown
8
10
  License-File: LICENSE
11
+ Requires-Dist: PyYAML
9
12
  Requires-Dist: requests
10
- Requires-Dist: pyyaml
13
+ Requires-Dist: setuptools
11
14
  Requires-Dist: termcolor
12
15
  Requires-Dist: psutil
13
16
  Requires-Dist: urlparse3
14
17
  Requires-Dist: tldextract
18
+ Requires-Dist: aiohttp
19
+ Dynamic: home-page
20
+ Dynamic: license-file
15
21
 
16
22
  <center><img src="https://github.com/xnl-h4ck3r/waymore/blob/main/waymore/images/title.png"></center>
17
23
 
18
- ## About - v4.5
24
+ ## About - v7.6
19
25
 
20
- The idea behind **waymore** is to find even more links from the Wayback Machine than other existing tools.
26
+ The idea behind **waymore** is to find even more links from the Wayback Machine (plus other sources) than other existing tools.
21
27
 
22
- 👉 The biggest difference between **waymore** and other tools is that it can also **download the archived responses** for URLs on wayback machine so that you can then search these for even more links, developer comments, extra parameters, etc. etc.
23
- 👉 Also, other tools do not currenrtly deal with the rate limiting now in place by the sources, and will often just stop with incomplete results and not let you know they are incomplete.
28
+ 👉 The biggest difference between **waymore** and other tools is that it can also **download the archived responses** for URLs on wayback machine (and URLScan) so that you can then search these for even more links, developer comments, extra parameters, etc. etc.
29
+ 👉 Also, other tools do not currently deal with the rate limiting now in place by the sources, and will often just stop with incomplete results and not let you know they are incomplete.
24
30
 
25
31
  Anyone who does bug bounty will have likely used the amazing [waybackurls](https://github.com/tomnomnom/waybackurls) by @TomNomNoms. This tool gets URLs from [web.archive.org](https://web.archive.org) and additional links (if any) from one of the index collections on [index.commoncrawl.org](http://index.commoncrawl.org/).
26
- You would have also likely used the amazing [gau](https://github.com/lc/gau) by @hacker\_ which also finds URL's from wayback archive, Common Crawl, but also from Alien Vault and URLScan.
32
+ You would have also likely used the amazing [gau](https://github.com/lc/gau) by @hacker\_ which also finds URL's from wayback archive, Common Crawl, but also from Alien Vault, URLScan, Virus Total and Intelligence X.
27
33
  Now **waymore** gets URL's from ALL of those sources too (with ability to filter more to get what you want):
28
34
 
29
35
  - Wayback Machine (web.archive.org)
@@ -31,6 +37,7 @@ Now **waymore** gets URL's from ALL of those sources too (with ability to filter
31
37
  - Alien Vault OTX (otx.alienvault.com)
32
38
  - URLScan (urlscan.io)
33
39
  - Virus Total (virustotal.com)
40
+ - Intelligence X (intelx.io) - ACADEMIA OR PAID TIERS ONLY
34
41
 
35
42
  👉 It's a point that many seem to miss, so I'll just add it again :) ... The biggest difference between **waymore** and other tools is that it can also **download the archived responses** for URLs on wayback machine so that you can then search these for even more links, developer comments, extra parameters, etc. etc.
36
43
 
@@ -44,7 +51,7 @@ Now **waymore** gets URL's from ALL of those sources too (with ability to filter
44
51
 
45
52
  **NOTE: If you already have a `config.yml` file, it will not be overwritten. The file `config.yml.NEW` will be created in the same directory. If you need the new config, remove `config.yml` and rename `config.yml.NEW` back to `config.yml`.**
46
53
 
47
- `waymore` supports **Python 3**.
54
+ `waymore` supports **Python 3.7+** (Python 3.7 or higher required for async/await support).
48
55
 
49
56
  Install `waymore` in default (global) python environment.
50
57
 
@@ -83,10 +90,12 @@ pipx install git+https://github.com/xnl-h4ck3r/waymore.git
83
90
  | -n | --no-subs | Don't include subdomains of the target domain (only used if input is not a domain with a specific path). |
84
91
  | -f | --filter-responses-only | The initial links from sources will not be filtered, only the responses that are downloaded, e.g. it maybe useful to still see all available paths from the links, even if you don't want to check the content. |
85
92
  | -fc | | Filter HTTP status codes for retrieved URLs and responses. Comma separated list of codes (default: the `FILTER_CODE` values from `config.yml`). Passing this argument will override the value from `config.yml` |
93
+ | -ft | | Filter MIME Types for retrieved URLs and responses. Comma separated list of MIME Types (default: the `FILTER_MIME` values from `config.yml`). Passing this argument will override the value from `config.yml`. **NOTE: This will NOT be applied to Alien Vault OTX, Virus Total and Intelligence X because they don't have the ability to filter on MIME Type. Sometimes URLScan does not have a MIME Type defined - these will always be included. Consider excluding sources if this matters to you.**. |
86
94
  | -mc | | Only Match HTTP status codes for retrieved URLs and responses. Comma separated list of codes. Passing this argument overrides the config `FILTER_CODE` and `-fc`. |
95
+ | -mt | | Only MIME Types for retrieved URLs and responses. Comma separated list of MIME types. Passing this argument overrides the config `FILTER_MIME` and `-ft`. **NOTE: This will NOT be applied to Alien Vault OTX, Virus Total and Intelligence X because they don't have the ability to filter on MIME Type. Sometimes URLScan does not have a MIME Type defined - these will always be included. Consider excluding sources if this matters to you.**. |
87
96
  | -l | --limit | How many responses will be saved (if `-mode R` or `-mode B` is passed). A positive value will get the **first N** results, a negative value will get the **last N** results. A value of 0 will get **ALL** responses (default: 5000) |
88
- | -from | --from-date | What date to get responses from. If not specified it will get from the earliest possible results. A partial value can be passed, e.g. `2016`, `201805`, etc. |
89
- | -to | --to-date | What date to get responses to. If not specified it will get to the latest possible results. A partial value can be passed, e.g. `2021`, `202112`, etc. |
97
+ | -from | --from-date | What date to get data from. If not specified it will get from the earliest possible results. A partial value can be passed, e.g. `2016`, `201805`, etc. **IMPORTANT: There are some exceptions with sources unable to get URLs within date limits: Virus Total - all known sub domains will still be returned; Intelligence X - all URLs will still be returned.** |
98
+ | -to | --to-date | What date to get data to. If not specified it will get to the latest possible results. A partial value can be passed, e.g. `2021`, `202112`, etc. **IMPORTANT: There are some exceptions with sources unable to get URLs within date limits: Virus Total - all known sub domains will still be returned; Intelligence X - all URLs will still be returned.** |
90
99
  | -ci | --capture-interval | Filters the search on archive.org to only get at most 1 capture per hour (`h`), day (`d`) or month (`m`). This filter is used for responses only. The default is `d` but can also be set to `none` to not filter anything and get all responses. |
91
100
  | -ra | --regex-after | RegEx for filtering purposes against links found from all sources of URLs AND responses downloaded. Only positive matches will be output. |
92
101
  | -url-filename | | Set the file name of downloaded responses to the URL that generated the response, otherwise it will be set to the hash value of the response. Using the hash value means multiple URLs that generated the same response will only result in one file being saved for that response. |
@@ -95,21 +104,24 @@ pipx install git+https://github.com/xnl-h4ck3r/waymore.git
95
104
  | -xav | | Exclude checks for links from alienvault.com |
96
105
  | -xus | | Exclude checks for links from urlscan.io |
97
106
  | -xvt | | Exclude checks for links from virustotal.com |
98
- | -lcc | | Limit the number of Common Crawl index collections searched, e.g. `-lcc 10` will just search the latest `10` collections (default: 3). As of July 2023 there are currently 95 collections. Setting to `0` (default) will search **ALL** collections. If you don't want to search Common Crawl at all, use the `-xcc` option. |
99
- | -lcy | | Limit the number of Common Crawl index collections searched by the year of the index data. The earliest index has data from 2008. Setting to 0 (default) will search collections or any year (but in conjuction with `-lcc`). For example, if you are only interested in data from 2015 and after, pass `-lcy 2015`. This will override the value of `-lcc` if passed. If you don't want to search Common Crawl at all, use the `-xcc` option. |
107
+ | -xix | | Exclude checks for links from Intelligence X.com |
108
+ | -lcc | | Limit the number of Common Crawl index collections searched, e.g. `-lcc 10` will just search the latest `10` collections (default: 1). As of November 2024 there are currently 106 collections. Setting to `0` will search **ALL** collections. If you don't want to search Common Crawl at all, use the `-xcc` option. |
100
109
  | -t | --timeout | This is for archived responses only! How many seconds to wait for the server to send data before giving up (default: 30) |
101
- | -p | --processes | Basic multithreading is done when getting requests for a file of URLs. This argument determines the number of processes (threads) used (default: 1) |
110
+ | -p | --processes | Basic multithreading is done when getting requests for a file of URLs. This argument determines the number of processes (threads) used (default: 2) |
102
111
  | -r | --retries | The number of retries for requests that get connection error or rate limited (default: 1). |
112
+ | -sip | --source-ip OR --bind-ip | Bind outbound HTTP/HTTPS requests to this source IP (useful on multi-homed hosts). Passing this argument overrides the `SOURCE_IP` value in the `config.yml` file. |
103
113
  | -m | --memory-threshold | The memory threshold percentage. If the machines memory goes above the threshold, the program will be stopped and ended gracefully before running out of memory (default: 95) |
104
114
  | -ko | --keywords-only | Only return links and responses that contain keywords that you are interested in. This can reduce the time it takes to get results. If you provide the flag with no value, Keywords are taken from the comma separated list in the `config.yml` file (typically in `~/.config/waymore/`) with the `FILTER_KEYWORDS` key, otherwise you can pass a specific Regex value to use, e.g. `-ko "admin"` to only get links containing the word `admin`, or `-ko "\.js(\?\|$)"` to only get JS files. The Regex check is NOT case sensitive. |
105
115
  | -lr | --limit-requests | Limit the number of requests that will be made when getting links from a source (this doesn\'t apply to Common Crawl). Some targets can return a huge amount of requests needed that are just not feasible to get, so this can be used to manage that situation. This defaults to 0 (Zero) which means there is no limit. |
106
116
  | -ow | --output-overwrite | If the URL output file (default `waymore.txt`, or specified by `-oU`) already exists, it will be overwritten instead of being appended to. |
107
117
  | -nlf | --new-links-file | If this argument is passed, a `waymore.new` file (or if `-oU` is used it will be the name of that file suffixed with `.new`) will also be written, and will contain links for the latest run. This can be used for continuous monitoring of a target (only for `mode U`, not `mode R`). |
118
+ | | --stream | Output URLs to STDOUT as soon as they are found (duplicates will be shown). Only works with `-mode U`. All other output is suppressed, so use `-v` to see any errors. Use `-oU` to explicitly save results to file (wil be deduplicated). |
108
119
  | -c | --config | Path to the YML config file. If not passed, it looks for file `config.yml` in the default directory, typically `~/.config/waymore`. |
109
120
  | -wrlr | --wayback-rate-limit-retry | The number of minutes the user wants to wait for a rate limit pause on Wayback Machine (archive.org) instead of stopping with a `429` error (default: 3). |
110
121
  | -urlr | --urlscan-rate-limit-retry | The number of minutes the user wants to wait for a rate limit pause on URLScan.io instead of stopping with a `429` error (default: 1). |
111
- | -co | --check-only | This will make a few minimal requests to show you how many requests, and roughly how long it could take, to get URLs from the sources and downloaded responses from Wayback Machine. |
122
+ | -co | --check-only | This will make a few minimal requests to show you how many requests, and roughly how long it could take, to get URLs from the sources and downloaded responses from Wayback Machine (unfortunately it isn't possible to check how long it will take to download responses from URLScan). |
112
123
  | -nd | --notify-discord | Whether to send a notification to Discord when waymore completes. It requires `WEBHOOK_DISCORD` to be provided in the `config.yml` file. |
124
+ | -nt | --notify-telegram | Whether to send a notification to Telegram when waymore completes. It requires `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID` to be provided in the `config.yml` file. |
113
125
  | -oijs | --output-inline-js | Whether to save combined inline javascript of all relevant files in the response directory when `-mode R` (or `-mode B`) has been used. The files are saved with the name `combinedInline{}.js` where `{}` is the number of the file, saving 1000 unique scripts per file. The file `combinedInlineSrc.txt` will also be created, containing the `src` value of all external scripts referenced in the files. |
114
126
  | -v | --verbose | Verbose output |
115
127
  | | --version | Show current version number. |
@@ -133,7 +145,7 @@ docker build -t waymore .
133
145
  Run waymore with this command:
134
146
 
135
147
  ```bash
136
- docker run -it --rm -v $PWD/results:/app/results waymore:latest waymore -i example.com -oU example.com.links -oR results/example.com/
148
+ docker run -it --rm -v $PWD/results:/app/results waymore:latest -i example.com -oU example.com.links -oR results/example.com/
137
149
  ```
138
150
 
139
151
  ## Input and Mode
@@ -154,16 +166,20 @@ If the input is just a domain, e.g. `redbull.com` then the `-mode` defaults to `
154
166
 
155
167
  The `config.yml` file (typically in `~/.config/waymore/`) have values that can be updated to suit your needs. Filters are all provided as comma separated lists:
156
168
 
157
- - `FILTER_CODE` - Exclusions used to exclude responses we will try to get from web.archive.org, and also for file names when `-i` is a directory, e.g. `301,302`. This can be overridden with the `-fc` argument. Passing the `-mc` (to match status codes instead of filter) will override any value in `FILTER_CODE` or `-fc`
158
- - `FILTER_MIME` - MIME Content-Type exclusions used to filter links and responses from web.archive.org through their API, e.g. `'text/css,image/jpeg`
169
+ - `FILTER_CODE` - Exclusions used to exclude responses we will try to get from web.archive.org, and also for file names when `-i` is a directory, e.g. `301,302`. This can be overridden with the `-fc` argument. Passing the `-mc` (to match status codes instead of filter) will override any value in `FILTER_CODE` or `-fc`.
170
+ - `FILTER_MIME` - MIME Content-Type exclusions used to filter links and responses from web.archive.org through their API, e.g. `'text/css,image/jpeg`. This can be overridden with the `-ft` argument. . Passing the `-mt` (to match MIME types instead of filter) will override any value in `FILTER_MIME` or `-ft`.
159
171
  - `FILTER_URL` - Response code exclusions we will use to filter links and responses from web.archive.org through their API, e.g. `.css,.jpg`
160
172
  - `FILTER_KEYWORDS` - Only links and responses will be returned that contain the specified keywords if the `-ko`/`--keywords-only` argument is passed (without providing an explicit value on the command line), e.g. `admin,portal`
161
173
  - `URLSCAN_API_KEY` - You can sign up to [urlscan.io](https://urlscan.io/user/signup) to get a **FREE** API key (there are also paid subscriptions available). It is recommended you get a key and put it into the config file so that you can get more back (and quicker) from their API. NOTE: You will get rate limited unless you have a full paid subscription.
162
174
  - `CONTINUE_RESPONSES_IF_PIPED` - If retrieving archive responses doesn't complete, you will be prompted next time whether you want to continue with the previous run. However, if `stdout` is piped to another process it is assumed you don't want to have an interactive prompt. A value of `True` (default) will determine assure the previous run will be continued. if you want a fresh run every time then set to `False`.
163
- - `WEBHOOK_DISCORD` - If the `--notify-discord` argument is passed, `knoxnl` will send a notification to this Discord wehook when a successful XSS is found.
175
+ - `WEBHOOK_DISCORD` - If the `--notify-discord` argument is passed, `waymore` will send a notification to this Discord wehook.
176
+ - `TELEGRAM_BOT_TOKEN` - If the `--notify-telegram` argument is passed, `waymore` will use this token to send a notification to Telegram.
177
+ - `TELEGRAM_CHAT_ID` - If the `--notify-telegram` argument is passed, `waymore` will send the notification to this chat ID.
164
178
  - `DEFAULT_OUTPUT_DIR` - This is the default location of any output files written if the `-oU` and `-oR` arguments are not used. If the value of this key is blank, then it will default to the location of the `config.yml` file.
179
+ - `INTELX_API_KEY` - You can sign up to [intelx.io here](https://intelx.io/product). It requires an academia or paid API key to do the `/phonebook/search` through their API (as of 2024-09-01, the Phonebook service has been restricted to academia or paid users due to constant abuse by spam accounts). You can get a free API key for academic use if you sign up with a valid academic email address.
180
+ - `SOURCE_IP` - Optional. Bind outbound HTTP/HTTPS requests to a specific source IP on multi-homed hosts. Can also be set via the `--source-ip/--bind-ip` CLI flag (CLI takes precedence).
165
181
 
166
- **NOTE: The MIME types cannot be filtered for Alien Vault results because they do not return that in the API response.**
182
+ **NOTE: The MIME types cannot be filtered for Alien Vault OTX, Virus Total and Intelligence X because they don't have the ability to filter on MIME Type. Sometimes URLScan does not have a MIME Type defined for a URL. In these cases, URLs will be included regardless of filter or match. Bear this in mind and consider excluding certain providers if this is important.**
167
183
 
168
184
  ## Output
169
185
 
@@ -201,6 +217,8 @@ The archive.org Wayback Machine CDX API can sometimes can sometimes require a hu
201
217
 
202
218
  There is also a problem with the Wayback Machine CDX API where the number of pages returned is not correct when filters are applied and can cause issues (see https://github.com/internetarchive/wayback/issues/243). Until that issue is resolved, setting the `-lr` argument to a sensible value can help with that problem in the short term.
203
219
 
220
+ The Common Crawl API has had a lot of issues for a long time. Including this source could make waymore take a lot longer to run and may not yield any extra results. You can check if tere is an issue by visiting http://index.commoncrawl.org/collinfo.json and seeing if this is successful. Consider excluding Common Crawl altogether using the `--providers` argument and not including `commoncrawl`, or using the `-xcc` argument.
221
+
204
222
  **The provider API servers aren't designed to cope with huge volumes, so be sensible and considerate about what you hit them with!**
205
223
 
206
224
  When downloading archived responses, this can take a long time and can sometimes be killed by the machine for some reason, or manually killed by the user.
@@ -218,7 +236,7 @@ The URLs are saved in the same path as `config.yml` (typically `~/.config/waymor
218
236
 
219
237
  ### Example 2
220
238
 
221
- Get ALL the URLs from Wayback for `redbull.com` (no filters are applied in `mode U` with `-f`, and no URLs are retrieved from Commone Crawl, Alien Vault, URLScan and Virus Total, because `-xcc`, `-xav`, `-xus`, `-xvt` are passed respectively).
239
+ Get ALL the URLs from Wayback for `redbull.com` (no filters are applied in `mode U` with `-f`, and no URLs are retrieved from Commone Crawl, Alien Vault, URLScan and Virus Total, because `-xcc`, `-xav`, `-xus`, `-xvt` are passed respectively. This can also be achieved by passing `--providers wayback` instead of the exclude arguments).
222
240
  Save the FIRST 200 responses that are found starting from 2022 (`-l 200 -from 2022`):
223
241
 
224
242
  <center><img src="https://github.com/xnl-h4ck3r/waymore/blob/main/waymore/images/example2.png"></center>
@@ -271,13 +289,23 @@ xnLinkFinder -i ~/Tools/waymore/results/redbull.com -sp https://www.redbull.com
271
289
 
272
290
  Or run other tools such as [trufflehog](https://github.com/trufflesecurity/trufflehog) or [gf](https://github.com/tomnomnom/gf) over the directory of responses to find even more from the archived responses!
273
291
 
292
+ ## In Depth Instructions
293
+
294
+ Below is an in-depth talk I did for [Jason Haddix's discord channel](https://discord.gg/jhaddix) back in March 2024 to cover **EVERYTHING** you need to know about `waymore`.
295
+
296
+ **NOTE: This video is from March 2024, so any features added after this will not be featured and some features may have changed. Please double check the current instructions.**
297
+
298
+ [![waymore talk](https://img.youtube.com/vi/hMaYSi9ErnM/0.jpg)](https://www.youtube.com/watch?v=hMaYSi9ErnM)
299
+
274
300
  ## Issues
275
301
 
276
302
  If you come across any problems at all, or have ideas for improvements, please feel free to raise an issue on Github. If there is a problem, it will be useful if you can provide the exact command you ran and a detailed description of the problem. If possible, run with `-v` to reproduce the problem and let me know about any error messages that are given.
277
303
 
278
304
  ## TODO
279
305
 
280
- - Add an `-oss` argument that accepts a file of Out Of Scope subdomains/URLs that will not be returned in the output, or have any responses downloaded
306
+ - Add an `-oos` argument that accepts a file of Out Of Scope subdomains/URLs that will not be returned in the output, or have any responses downloaded.
307
+ - The `waymore_index.txt` isn't de-duplicated if run multiple times for the same input with `-mode R` or `-mode B`.
308
+ - Rewrite to get from sources in parallel. Currently they are run consecutively sorry!
281
309
 
282
310
  ## References
283
311
 
@@ -286,6 +314,7 @@ If you come across any problems at all, or have ideas for improvements, please f
286
314
  - [Alien Vault OTX API](https://otx.alienvault.com/assets/static/external_api.html)
287
315
  - [URLScan API](https://urlscan.io/docs/api/)
288
316
  - [VirusTotal API (v2)](https://docs.virustotal.com/v2.0/reference/getting-started)
317
+ - [Intelligence X SDK](https://github.com/IntelligenceX/SDK?tab=readme-ov-file#intelligence-x-public-sdk)
289
318
 
290
319
  Good luck and good hunting!
291
320
  If you really love the tool (or any others), or they helped you find an awesome bounty, consider [BUYING ME A COFFEE!](https://ko-fi.com/xnlh4ck3r) ☕ (I could use the caffeine!)