email_data 1601156760
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.github/FUNDING.yml +3 -0
- data/.gitignore +10 -0
- data/.rubocop.yml +7 -0
- data/.travis.yml +6 -0
- data/CODE_OF_CONDUCT.md +74 -0
- data/Gemfile +6 -0
- data/LICENSE.txt +21 -0
- data/README.md +90 -0
- data/Rakefile +12 -0
- data/VERSION +1 -0
- data/bin/console +14 -0
- data/bin/helpers.rb +229 -0
- data/bin/setup +8 -0
- data/bin/sync +7 -0
- data/bin/sync-disposable-emails +213 -0
- data/bin/sync-tld +28 -0
- data/data/country_tlds.txt +235 -0
- data/data/disposable_domains.txt +111121 -0
- data/data/disposable_emails.txt +38 -0
- data/data/free_email_domains.txt +111 -0
- data/data/manual/disposable_domains.txt +0 -0
- data/data/manual/disposable_emails.txt +0 -0
- data/data/manual/free_email_domains.txt +111 -0
- data/data/tlds.txt +1508 -0
- data/email_data.gemspec +38 -0
- data/lib/email_data.rb +39 -0
- data/lib/email_data/version.rb +5 -0
- metadata +184 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: f7a8f3df2f2943f47d66fb39399eb771a4d95068818fd8569bdee6455b057894
|
|
4
|
+
data.tar.gz: 0b103a3048fae80b575687af7c2aea172ebc43fc6237041edd8b97f04e0b240e
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: b076e2437aca0b2e94dc60533f703e358b5796e254100c37f1d2fd4dbc5f07b5ceb7d167685aed004ada35f4ae5d26b1ca28ccbcf494a147858cd40f4b7efbae
|
|
7
|
+
data.tar.gz: d2ac1858ed07ad527416a434f9923e292139dcfbbe97ea78068f270d1049516a90e81cb02a50101b66d17ada0829b7ca0a77d6d7f07980498a1a54094f08baeb
|
data/.github/FUNDING.yml
ADDED
data/.gitignore
ADDED
data/.rubocop.yml
ADDED
data/.travis.yml
ADDED
data/CODE_OF_CONDUCT.md
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# Contributor Covenant Code of Conduct
|
|
2
|
+
|
|
3
|
+
## Our Pledge
|
|
4
|
+
|
|
5
|
+
In the interest of fostering an open and welcoming environment, we as
|
|
6
|
+
contributors and maintainers pledge to making participation in our project and
|
|
7
|
+
our community a harassment-free experience for everyone, regardless of age, body
|
|
8
|
+
size, disability, ethnicity, gender identity and expression, level of experience,
|
|
9
|
+
nationality, personal appearance, race, religion, or sexual identity and
|
|
10
|
+
orientation.
|
|
11
|
+
|
|
12
|
+
## Our Standards
|
|
13
|
+
|
|
14
|
+
Examples of behavior that contributes to creating a positive environment
|
|
15
|
+
include:
|
|
16
|
+
|
|
17
|
+
* Using welcoming and inclusive language
|
|
18
|
+
* Being respectful of differing viewpoints and experiences
|
|
19
|
+
* Gracefully accepting constructive criticism
|
|
20
|
+
* Focusing on what is best for the community
|
|
21
|
+
* Showing empathy towards other community members
|
|
22
|
+
|
|
23
|
+
Examples of unacceptable behavior by participants include:
|
|
24
|
+
|
|
25
|
+
* The use of sexualized language or imagery and unwelcome sexual attention or
|
|
26
|
+
advances
|
|
27
|
+
* Trolling, insulting/derogatory comments, and personal or political attacks
|
|
28
|
+
* Public or private harassment
|
|
29
|
+
* Publishing others' private information, such as a physical or electronic
|
|
30
|
+
address, without explicit permission
|
|
31
|
+
* Other conduct which could reasonably be considered inappropriate in a
|
|
32
|
+
professional setting
|
|
33
|
+
|
|
34
|
+
## Our Responsibilities
|
|
35
|
+
|
|
36
|
+
Project maintainers are responsible for clarifying the standards of acceptable
|
|
37
|
+
behavior and are expected to take appropriate and fair corrective action in
|
|
38
|
+
response to any instances of unacceptable behavior.
|
|
39
|
+
|
|
40
|
+
Project maintainers have the right and responsibility to remove, edit, or
|
|
41
|
+
reject comments, commits, code, wiki edits, issues, and other contributions
|
|
42
|
+
that are not aligned to this Code of Conduct, or to ban temporarily or
|
|
43
|
+
permanently any contributor for other behaviors that they deem inappropriate,
|
|
44
|
+
threatening, offensive, or harmful.
|
|
45
|
+
|
|
46
|
+
## Scope
|
|
47
|
+
|
|
48
|
+
This Code of Conduct applies both within project spaces and in public spaces
|
|
49
|
+
when an individual is representing the project or its community. Examples of
|
|
50
|
+
representing a project or community include using an official project e-mail
|
|
51
|
+
address, posting via an official social media account, or acting as an appointed
|
|
52
|
+
representative at an online or offline event. Representation of a project may be
|
|
53
|
+
further defined and clarified by project maintainers.
|
|
54
|
+
|
|
55
|
+
## Enforcement
|
|
56
|
+
|
|
57
|
+
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
|
58
|
+
reported by contacting the project team at me@fnando.com. All
|
|
59
|
+
complaints will be reviewed and investigated and will result in a response that
|
|
60
|
+
is deemed necessary and appropriate to the circumstances. The project team is
|
|
61
|
+
obligated to maintain confidentiality with regard to the reporter of an incident.
|
|
62
|
+
Further details of specific enforcement policies may be posted separately.
|
|
63
|
+
|
|
64
|
+
Project maintainers who do not follow or enforce the Code of Conduct in good
|
|
65
|
+
faith may face temporary or permanent repercussions as determined by other
|
|
66
|
+
members of the project's leadership.
|
|
67
|
+
|
|
68
|
+
## Attribution
|
|
69
|
+
|
|
70
|
+
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
|
|
71
|
+
available at [https://contributor-covenant.org/version/1/4][version]
|
|
72
|
+
|
|
73
|
+
[homepage]: https://contributor-covenant.org
|
|
74
|
+
[version]: https://contributor-covenant.org/version/1/4/
|
data/Gemfile
ADDED
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2020 Nando Vieira
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
|
13
|
+
all copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# EmailData
|
|
2
|
+
|
|
3
|
+
This project is a compilation of datasets related to emails.
|
|
4
|
+
|
|
5
|
+
- Disposable emails
|
|
6
|
+
- Disposable domains
|
|
7
|
+
- Free email services
|
|
8
|
+
|
|
9
|
+
The data is compiled from several different sources, so thank you all for making
|
|
10
|
+
this data available.
|
|
11
|
+
|
|
12
|
+
## Installation
|
|
13
|
+
|
|
14
|
+
Add this line to your application's Gemfile:
|
|
15
|
+
|
|
16
|
+
```ruby
|
|
17
|
+
gem "email_data"
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
And then execute:
|
|
21
|
+
|
|
22
|
+
$ bundle install
|
|
23
|
+
|
|
24
|
+
Or install it yourself as:
|
|
25
|
+
|
|
26
|
+
$ gem install email_data
|
|
27
|
+
|
|
28
|
+
## Usage
|
|
29
|
+
|
|
30
|
+
```ruby
|
|
31
|
+
require "email_data"
|
|
32
|
+
|
|
33
|
+
# <Pathname /> instance pointing to the data directory.
|
|
34
|
+
EmailData.data_dir
|
|
35
|
+
|
|
36
|
+
# List of disposable domains. Punycode is expanded into ASCII domains.
|
|
37
|
+
EmailData.disposable_domains
|
|
38
|
+
|
|
39
|
+
# List of disposable emails. Some services use free email like Gmail to create
|
|
40
|
+
# disposable emails.
|
|
41
|
+
EmailData.disposable_emails
|
|
42
|
+
|
|
43
|
+
# List of free email services.
|
|
44
|
+
EmailData.free_email_domains
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Dataset
|
|
48
|
+
|
|
49
|
+
The dataset is updated automatically. If you have any manual entries you would
|
|
50
|
+
like to add, please make a pull request against the files `data/manual/*.txt`.
|
|
51
|
+
|
|
52
|
+
- `data/manual/disposable_domains.txt`: only domains from disposable servers
|
|
53
|
+
must go here.
|
|
54
|
+
- `data/manual/disposable_emails.txt`: only normalized email addresses that use
|
|
55
|
+
free email services must go here. E.g. `d.i.s.p.o.s.a.b.l.e+1234@gmail.com`
|
|
56
|
+
must be added as `disposable@gmail.com`.
|
|
57
|
+
- `data/manual/free_email_domains.txt`: only free email services must go here.
|
|
58
|
+
These are services that allow anyone to create an email account, even if it's
|
|
59
|
+
just a trial without credit cards.
|
|
60
|
+
|
|
61
|
+
## Development
|
|
62
|
+
|
|
63
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run
|
|
64
|
+
`rake test` to run the tests. You can also run `bin/console` for an interactive
|
|
65
|
+
prompt that will allow you to experiment.
|
|
66
|
+
|
|
67
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To
|
|
68
|
+
release a new version, update the version number in `version.rb`, and then run
|
|
69
|
+
`bundle exec rake release`, which will create a git tag for the version, push
|
|
70
|
+
git commits and tags, and push the `.gem` file to
|
|
71
|
+
[rubygems.org](https://rubygems.org).
|
|
72
|
+
|
|
73
|
+
## Contributing
|
|
74
|
+
|
|
75
|
+
Bug reports and pull requests are welcome on GitHub at
|
|
76
|
+
https://github.com/fnando/email_data. This project is intended to be a safe,
|
|
77
|
+
welcoming space for collaboration, and contributors are expected to adhere to
|
|
78
|
+
the
|
|
79
|
+
[code of conduct](https://github.com/fnando/email_data/blob/main/CODE_OF_CONDUCT.md).
|
|
80
|
+
|
|
81
|
+
## License
|
|
82
|
+
|
|
83
|
+
The gem is available as open source under the terms of the
|
|
84
|
+
[MIT License](https://opensource.org/licenses/MIT).
|
|
85
|
+
|
|
86
|
+
## Code of Conduct
|
|
87
|
+
|
|
88
|
+
Everyone interacting in the EmailData project's codebases, issue trackers, chat
|
|
89
|
+
rooms and mailing lists is expected to follow the
|
|
90
|
+
[code of conduct](https://github.com/fnando/email_data/blob/main/CODE_OF_CONDUCT.md).
|
data/Rakefile
ADDED
data/VERSION
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
1601156760
|
data/bin/console
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
|
|
3
|
+
require "bundler/setup"
|
|
4
|
+
require "email_data"
|
|
5
|
+
|
|
6
|
+
# You can add fixtures and/or initialization code here to make experimenting
|
|
7
|
+
# with your gem easier. You can also use a different console, if you like.
|
|
8
|
+
|
|
9
|
+
# (If you use this, don't forget to add pry to your Gemfile!)
|
|
10
|
+
# require "pry"
|
|
11
|
+
# Pry.start
|
|
12
|
+
|
|
13
|
+
require "irb"
|
|
14
|
+
IRB.start(__FILE__)
|
data/bin/helpers.rb
ADDED
|
@@ -0,0 +1,229 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
Encoding.default_internal = Encoding.find("UTF-8")
|
|
4
|
+
Encoding.default_external = Encoding.find("UTF-8")
|
|
5
|
+
|
|
6
|
+
Thread.abort_on_exception = false
|
|
7
|
+
Thread.report_on_exception = false
|
|
8
|
+
|
|
9
|
+
require "open-uri"
|
|
10
|
+
require "json"
|
|
11
|
+
require "aitch"
|
|
12
|
+
require "logger"
|
|
13
|
+
require "simpleidn"
|
|
14
|
+
require "timeout"
|
|
15
|
+
require "active_support/all"
|
|
16
|
+
require "whois"
|
|
17
|
+
require "whois-parser"
|
|
18
|
+
require "root_domain"
|
|
19
|
+
|
|
20
|
+
USER_AGENT = [
|
|
21
|
+
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15",
|
|
22
|
+
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/1.0.154.53 Safari/525.19",
|
|
23
|
+
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/1.0.154.36 Safari/525.19",
|
|
24
|
+
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/7.0.540.0 Safari/534.10",
|
|
25
|
+
"Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/534.4 (KHTML, like Gecko) Chrome/6.0.481.0 Safari/534.4",
|
|
26
|
+
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.86 Safari/533.4",
|
|
27
|
+
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.2 (KHTML, like Gecko) Chrome/4.0.223.3 Safari/532.2",
|
|
28
|
+
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/4.0.201.1 Safari/532.0",
|
|
29
|
+
"Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.195.27 Safari/532.0",
|
|
30
|
+
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.173.1 Safari/530.5",
|
|
31
|
+
"Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.558.0 Safari/534.10",
|
|
32
|
+
"Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/540.0 (KHTML,like Gecko) Chrome/9.1.0.0 Safari/540.0",
|
|
33
|
+
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/9.0.600.0 Safari/534.14",
|
|
34
|
+
"Mozilla/5.0 (X11; U; Windows NT 6; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12",
|
|
35
|
+
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.0 Safari/534.13",
|
|
36
|
+
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.11 Safari/534.16",
|
|
37
|
+
"Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20",
|
|
38
|
+
"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.792.0 Safari/535.1",
|
|
39
|
+
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.872.0 Safari/535.2",
|
|
40
|
+
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7",
|
|
41
|
+
"Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.66 Safari/535.11",
|
|
42
|
+
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.45 Safari/535.19",
|
|
43
|
+
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
|
|
44
|
+
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
|
|
45
|
+
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
|
|
46
|
+
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko) Chrome/24.0.1295.0 Safari/537.15",
|
|
47
|
+
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36",
|
|
48
|
+
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1467.0 Safari/537.36",
|
|
49
|
+
"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36",
|
|
50
|
+
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1623.0 Safari/537.36",
|
|
51
|
+
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36",
|
|
52
|
+
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36",
|
|
53
|
+
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.38 Safari/537.36",
|
|
54
|
+
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36",
|
|
55
|
+
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
|
|
56
|
+
"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36",
|
|
57
|
+
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
|
|
58
|
+
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 GTB5",
|
|
59
|
+
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; ko; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2",
|
|
60
|
+
"Mozilla/5.0 (X11; U; SunOS sun4u; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5",
|
|
61
|
+
"Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.12) Gecko/20080214 Firefox/2.0.0.12",
|
|
62
|
+
"Mozilla/5.0 (Windows; U; Windows NT 5.1; cs; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8",
|
|
63
|
+
"Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.8.0.5) Gecko/20060819 Firefox/1.5.0.5",
|
|
64
|
+
"Mozilla/5.0 (Windows; U; Windows NT 5.0; es-ES; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3",
|
|
65
|
+
"Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5",
|
|
66
|
+
"Mozilla/5.0 (Windows; Windows NT 6.1; rv:2.0b2) Gecko/20100720 Firefox/4.0b2",
|
|
67
|
+
"Mozilla/5.0 (X11; Linux x86_64; rv:2.0b4) Gecko/20100818 Firefox/4.0b4",
|
|
68
|
+
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2) Gecko/20100308 Ubuntu/10.04 (lucid) Firefox/3.6 GTB7.1",
|
|
69
|
+
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b7) Gecko/20101111 Firefox/4.0b7",
|
|
70
|
+
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b8pre) Gecko/20101114 Firefox/4.0b8pre",
|
|
71
|
+
"Mozilla/5.0 (X11; Linux x86_64; rv:2.0b9pre) Gecko/20110111 Firefox/4.0b9pre",
|
|
72
|
+
"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b9pre) Gecko/20101228 Firefox/4.0b9pre",
|
|
73
|
+
"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.2a1pre) Gecko/20110324 Firefox/4.2a1pre",
|
|
74
|
+
"Mozilla/5.0 (X11; U; Linux amd64; rv:5.0) Gecko/20100101 Firefox/5.0 (Debian)",
|
|
75
|
+
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a2) Gecko/20110613 Firefox/6.0a2",
|
|
76
|
+
"Mozilla/5.0 (X11; Linux i686 on x86_64; rv:12.0) Gecko/20100101 Firefox/12.0",
|
|
77
|
+
"Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2",
|
|
78
|
+
"Mozilla/5.0 (X11; Ubuntu; Linux armv7l; rv:17.0) Gecko/20100101 Firefox/17.0",
|
|
79
|
+
"Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20130328 Firefox/21.0",
|
|
80
|
+
"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:22.0) Gecko/20130328 Firefox/22.0",
|
|
81
|
+
"Mozilla/5.0 (Windows NT 5.1; rv:25.0) Gecko/20100101 Firefox/25.0",
|
|
82
|
+
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:25.0) Gecko/20100101 Firefox/25.0",
|
|
83
|
+
"Mozilla/5.0 (Windows NT 6.1; rv:28.0) Gecko/20100101 Firefox/28.0",
|
|
84
|
+
"Mozilla/5.0 (X11; Linux i686; rv:30.0) Gecko/20100101 Firefox/30.0",
|
|
85
|
+
"Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0",
|
|
86
|
+
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0",
|
|
87
|
+
"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0",
|
|
88
|
+
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:58.0) Gecko/20100101 Firefox/58.0"
|
|
89
|
+
].freeze
|
|
90
|
+
|
|
91
|
+
Aitch.configure do |config|
|
|
92
|
+
# config.logger = Logger.new($stdout)
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
module Enumerable
|
|
96
|
+
def count_by
|
|
97
|
+
return to_enum(:count_by) unless block_given?
|
|
98
|
+
|
|
99
|
+
each_with_object(Hash.new(0)) do |item, buffer|
|
|
100
|
+
buffer[yield(item)] += 1
|
|
101
|
+
end
|
|
102
|
+
end
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
def random_timeout
|
|
106
|
+
(1..10).map {|i| (1.0 / i).round(2) }.sample
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
def refresh_list(url:, path:, verb: :get, params: nil, &block)
|
|
110
|
+
response = http_request(verb, url, params)
|
|
111
|
+
items = block.call(response)
|
|
112
|
+
append_to_file(path, items)
|
|
113
|
+
rescue StandardError => error
|
|
114
|
+
puts "=> [ERROR] Unable to refresh list: #{url} (#{error.class}: #{error.message})"
|
|
115
|
+
[]
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
def append_to_file(path, new_list)
|
|
119
|
+
list = load_file(path) || []
|
|
120
|
+
list += new_list
|
|
121
|
+
save_file(path, list)
|
|
122
|
+
list
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
def save_file(path, list)
|
|
126
|
+
list = list.map do |item|
|
|
127
|
+
if item.include?("@")
|
|
128
|
+
mailbox, domain = item.split("@")
|
|
129
|
+
else
|
|
130
|
+
domain = item
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
domain = domain
|
|
134
|
+
.squish
|
|
135
|
+
.gsub(/\s+/, "")
|
|
136
|
+
.downcase
|
|
137
|
+
|
|
138
|
+
[mailbox, SimpleIDN.to_ascii(domain)].compact.join("@")
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
list = list.uniq.sort
|
|
142
|
+
|
|
143
|
+
file_path = "#{__dir__}/../data/#{path}"
|
|
144
|
+
|
|
145
|
+
FileUtils.mkdir_p(File.dirname(file_path))
|
|
146
|
+
|
|
147
|
+
File.open(file_path, "w") do |file|
|
|
148
|
+
file << if path.end_with?(".json")
|
|
149
|
+
JSON.pretty_generate(list)
|
|
150
|
+
else
|
|
151
|
+
list.join("\n")
|
|
152
|
+
end
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
list
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
def load_file(path, default = [])
|
|
159
|
+
path = File.expand_path("#{__dir__}/../data/#{path}")
|
|
160
|
+
|
|
161
|
+
return default unless File.file?(path)
|
|
162
|
+
|
|
163
|
+
if path.end_with?(".json")
|
|
164
|
+
JSON.parse(File.read(path))
|
|
165
|
+
else
|
|
166
|
+
File.read(path).lines.map(&:chomp)
|
|
167
|
+
end
|
|
168
|
+
rescue StandardError => error
|
|
169
|
+
puts "=> [ERROR] Unable to load #{path} (#{error.class}: #{error.message})"
|
|
170
|
+
default
|
|
171
|
+
end
|
|
172
|
+
|
|
173
|
+
def http_request(verb, url, params = nil)
|
|
174
|
+
referer = url[%r{^(https?://[^/]+/)}, 1]
|
|
175
|
+
|
|
176
|
+
headers = {
|
|
177
|
+
"user-agent" => USER_AGENT.sample,
|
|
178
|
+
"X-Requested-With" => "XMLHttpRequest",
|
|
179
|
+
"Referer" => referer,
|
|
180
|
+
"Origin" => referer
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
Aitch.public_send(verb, url: url, headers: headers, params: params, options: {expect: 200})
|
|
184
|
+
end
|
|
185
|
+
|
|
186
|
+
def root_domains(domains)
|
|
187
|
+
processed_domains = domains.map do |domain|
|
|
188
|
+
RootDomain.call(domain)
|
|
189
|
+
end
|
|
190
|
+
|
|
191
|
+
processed_domains.reject(&:nil?)
|
|
192
|
+
end
|
|
193
|
+
|
|
194
|
+
def thread(&block)
|
|
195
|
+
Thread.new do
|
|
196
|
+
block.call
|
|
197
|
+
rescue StandardError => error
|
|
198
|
+
puts "=> [ERROR] Thread has errored; #{error.class}: #{error.message}"
|
|
199
|
+
end
|
|
200
|
+
end
|
|
201
|
+
|
|
202
|
+
def load_github_url(url)
|
|
203
|
+
puts "=> Fetching #{url}"
|
|
204
|
+
|
|
205
|
+
basename = URI.parse(url).path[%r{/([^/]+/[^/]+)}, 1].tr("/", "_").tr("-", "_")
|
|
206
|
+
path = "disposable/#{basename}.txt"
|
|
207
|
+
domains = load_file(path)
|
|
208
|
+
|
|
209
|
+
ext = File.extname(url)
|
|
210
|
+
|
|
211
|
+
domains += case ext
|
|
212
|
+
when ".json"
|
|
213
|
+
JSON.parse(http_request(:get, url).body)
|
|
214
|
+
when ".txt"
|
|
215
|
+
http_request(:get, url).body.lines.map(&:chomp)
|
|
216
|
+
else
|
|
217
|
+
raise "Unknown extension"
|
|
218
|
+
end
|
|
219
|
+
|
|
220
|
+
append_to_file(path, domains)
|
|
221
|
+
domains
|
|
222
|
+
rescue StandardError => error
|
|
223
|
+
puts "=> Unable to load #{url}; #{error.class}: #{error.message}"
|
|
224
|
+
[]
|
|
225
|
+
end
|
|
226
|
+
|
|
227
|
+
def normalize_list(list)
|
|
228
|
+
list.flatten.map(&:chomp).compact.reject(&:empty?)
|
|
229
|
+
end
|