email_data 1601156760

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: f7a8f3df2f2943f47d66fb39399eb771a4d95068818fd8569bdee6455b057894
4
+ data.tar.gz: 0b103a3048fae80b575687af7c2aea172ebc43fc6237041edd8b97f04e0b240e
5
+ SHA512:
6
+ metadata.gz: b076e2437aca0b2e94dc60533f703e358b5796e254100c37f1d2fd4dbc5f07b5ceb7d167685aed004ada35f4ae5d26b1ca28ccbcf494a147858cd40f4b7efbae
7
+ data.tar.gz: d2ac1858ed07ad527416a434f9923e292139dcfbbe97ea78068f270d1049516a90e81cb02a50101b66d17ada0829b7ca0a77d6d7f07980498a1a54094f08baeb
@@ -0,0 +1,3 @@
1
+ ---
2
+ github: [fnando]
3
+ custom: ["https://www.paypal.me/nandovieira/🍕"]
@@ -0,0 +1,10 @@
1
+ /.bundle/
2
+ /.yardoc
3
+ /_yardoc/
4
+ /coverage/
5
+ /doc/
6
+ /pkg/
7
+ /spec/reports/
8
+ /tmp/
9
+ /data/disposable/**/**
10
+ Gemfile.lock
@@ -0,0 +1,7 @@
1
+ ---
2
+ inherit_gem:
3
+ rubocop-fnando: .rubocop.yml
4
+
5
+ AllCops:
6
+ TargetRubyVersion: 2.7
7
+ NewCops: enable
@@ -0,0 +1,6 @@
1
+ ---
2
+ language: ruby
3
+ cache: bundler
4
+ rvm:
5
+ - 2.7.1
6
+ before_install: gem install bundler -v 2.1.4
@@ -0,0 +1,74 @@
1
+ # Contributor Covenant Code of Conduct
2
+
3
+ ## Our Pledge
4
+
5
+ In the interest of fostering an open and welcoming environment, we as
6
+ contributors and maintainers pledge to making participation in our project and
7
+ our community a harassment-free experience for everyone, regardless of age, body
8
+ size, disability, ethnicity, gender identity and expression, level of experience,
9
+ nationality, personal appearance, race, religion, or sexual identity and
10
+ orientation.
11
+
12
+ ## Our Standards
13
+
14
+ Examples of behavior that contributes to creating a positive environment
15
+ include:
16
+
17
+ * Using welcoming and inclusive language
18
+ * Being respectful of differing viewpoints and experiences
19
+ * Gracefully accepting constructive criticism
20
+ * Focusing on what is best for the community
21
+ * Showing empathy towards other community members
22
+
23
+ Examples of unacceptable behavior by participants include:
24
+
25
+ * The use of sexualized language or imagery and unwelcome sexual attention or
26
+ advances
27
+ * Trolling, insulting/derogatory comments, and personal or political attacks
28
+ * Public or private harassment
29
+ * Publishing others' private information, such as a physical or electronic
30
+ address, without explicit permission
31
+ * Other conduct which could reasonably be considered inappropriate in a
32
+ professional setting
33
+
34
+ ## Our Responsibilities
35
+
36
+ Project maintainers are responsible for clarifying the standards of acceptable
37
+ behavior and are expected to take appropriate and fair corrective action in
38
+ response to any instances of unacceptable behavior.
39
+
40
+ Project maintainers have the right and responsibility to remove, edit, or
41
+ reject comments, commits, code, wiki edits, issues, and other contributions
42
+ that are not aligned to this Code of Conduct, or to ban temporarily or
43
+ permanently any contributor for other behaviors that they deem inappropriate,
44
+ threatening, offensive, or harmful.
45
+
46
+ ## Scope
47
+
48
+ This Code of Conduct applies both within project spaces and in public spaces
49
+ when an individual is representing the project or its community. Examples of
50
+ representing a project or community include using an official project e-mail
51
+ address, posting via an official social media account, or acting as an appointed
52
+ representative at an online or offline event. Representation of a project may be
53
+ further defined and clarified by project maintainers.
54
+
55
+ ## Enforcement
56
+
57
+ Instances of abusive, harassing, or otherwise unacceptable behavior may be
58
+ reported by contacting the project team at me@fnando.com. All
59
+ complaints will be reviewed and investigated and will result in a response that
60
+ is deemed necessary and appropriate to the circumstances. The project team is
61
+ obligated to maintain confidentiality with regard to the reporter of an incident.
62
+ Further details of specific enforcement policies may be posted separately.
63
+
64
+ Project maintainers who do not follow or enforce the Code of Conduct in good
65
+ faith may face temporary or permanent repercussions as determined by other
66
+ members of the project's leadership.
67
+
68
+ ## Attribution
69
+
70
+ This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
71
+ available at [https://contributor-covenant.org/version/1/4][version]
72
+
73
+ [homepage]: https://contributor-covenant.org
74
+ [version]: https://contributor-covenant.org/version/1/4/
data/Gemfile ADDED
@@ -0,0 +1,6 @@
1
+ # frozen_string_literal: true
2
+
3
+ source "https://rubygems.org"
4
+
5
+ # Specify your gem's dependencies in email_data.gemspec
6
+ gemspec
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2020 Nando Vieira
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
@@ -0,0 +1,90 @@
1
+ # EmailData
2
+
3
+ This project is a compilation of datasets related to emails.
4
+
5
+ - Disposable emails
6
+ - Disposable domains
7
+ - Free email services
8
+
9
+ The data is compiled from several different sources, so thank you all for making
10
+ this data available.
11
+
12
+ ## Installation
13
+
14
+ Add this line to your application's Gemfile:
15
+
16
+ ```ruby
17
+ gem "email_data"
18
+ ```
19
+
20
+ And then execute:
21
+
22
+ $ bundle install
23
+
24
+ Or install it yourself as:
25
+
26
+ $ gem install email_data
27
+
28
+ ## Usage
29
+
30
+ ```ruby
31
+ require "email_data"
32
+
33
+ # <Pathname /> instance pointing to the data directory.
34
+ EmailData.data_dir
35
+
36
+ # List of disposable domains. Punycode is expanded into ASCII domains.
37
+ EmailData.disposable_domains
38
+
39
+ # List of disposable emails. Some services use free email like Gmail to create
40
+ # disposable emails.
41
+ EmailData.disposable_emails
42
+
43
+ # List of free email services.
44
+ EmailData.free_email_domains
45
+ ```
46
+
47
+ ## Dataset
48
+
49
+ The dataset is updated automatically. If you have any manual entries you would
50
+ like to add, please make a pull request against the files `data/manual/*.txt`.
51
+
52
+ - `data/manual/disposable_domains.txt`: only domains from disposable servers
53
+ must go here.
54
+ - `data/manual/disposable_emails.txt`: only normalized email addresses that use
55
+ free email services must go here. E.g. `d.i.s.p.o.s.a.b.l.e+1234@gmail.com`
56
+ must be added as `disposable@gmail.com`.
57
+ - `data/manual/free_email_domains.txt`: only free email services must go here.
58
+ These are services that allow anyone to create an email account, even if it's
59
+ just a trial without credit cards.
60
+
61
+ ## Development
62
+
63
+ After checking out the repo, run `bin/setup` to install dependencies. Then, run
64
+ `rake test` to run the tests. You can also run `bin/console` for an interactive
65
+ prompt that will allow you to experiment.
66
+
67
+ To install this gem onto your local machine, run `bundle exec rake install`. To
68
+ release a new version, update the version number in `version.rb`, and then run
69
+ `bundle exec rake release`, which will create a git tag for the version, push
70
+ git commits and tags, and push the `.gem` file to
71
+ [rubygems.org](https://rubygems.org).
72
+
73
+ ## Contributing
74
+
75
+ Bug reports and pull requests are welcome on GitHub at
76
+ https://github.com/fnando/email_data. This project is intended to be a safe,
77
+ welcoming space for collaboration, and contributors are expected to adhere to
78
+ the
79
+ [code of conduct](https://github.com/fnando/email_data/blob/main/CODE_OF_CONDUCT.md).
80
+
81
+ ## License
82
+
83
+ The gem is available as open source under the terms of the
84
+ [MIT License](https://opensource.org/licenses/MIT).
85
+
86
+ ## Code of Conduct
87
+
88
+ Everyone interacting in the EmailData project's codebases, issue trackers, chat
89
+ rooms and mailing lists is expected to follow the
90
+ [code of conduct](https://github.com/fnando/email_data/blob/main/CODE_OF_CONDUCT.md).
@@ -0,0 +1,12 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "bundler/gem_tasks"
4
+ require "rake/testtask"
5
+
6
+ Rake::TestTask.new(:test) do |t|
7
+ t.libs << "test"
8
+ t.libs << "lib"
9
+ t.test_files = FileList["test/**/*_test.rb"]
10
+ end
11
+
12
+ task default: :test
data/VERSION ADDED
@@ -0,0 +1 @@
1
+ 1601156760
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require "bundler/setup"
4
+ require "email_data"
5
+
6
+ # You can add fixtures and/or initialization code here to make experimenting
7
+ # with your gem easier. You can also use a different console, if you like.
8
+
9
+ # (If you use this, don't forget to add pry to your Gemfile!)
10
+ # require "pry"
11
+ # Pry.start
12
+
13
+ require "irb"
14
+ IRB.start(__FILE__)
@@ -0,0 +1,229 @@
1
+ # frozen_string_literal: true
2
+
3
+ Encoding.default_internal = Encoding.find("UTF-8")
4
+ Encoding.default_external = Encoding.find("UTF-8")
5
+
6
+ Thread.abort_on_exception = false
7
+ Thread.report_on_exception = false
8
+
9
+ require "open-uri"
10
+ require "json"
11
+ require "aitch"
12
+ require "logger"
13
+ require "simpleidn"
14
+ require "timeout"
15
+ require "active_support/all"
16
+ require "whois"
17
+ require "whois-parser"
18
+ require "root_domain"
19
+
20
+ USER_AGENT = [
21
+ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15",
22
+ "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/1.0.154.53 Safari/525.19",
23
+ "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/1.0.154.36 Safari/525.19",
24
+ "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/7.0.540.0 Safari/534.10",
25
+ "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/534.4 (KHTML, like Gecko) Chrome/6.0.481.0 Safari/534.4",
26
+ "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.86 Safari/533.4",
27
+ "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.2 (KHTML, like Gecko) Chrome/4.0.223.3 Safari/532.2",
28
+ "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/4.0.201.1 Safari/532.0",
29
+ "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.195.27 Safari/532.0",
30
+ "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.173.1 Safari/530.5",
31
+ "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.558.0 Safari/534.10",
32
+ "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/540.0 (KHTML,like Gecko) Chrome/9.1.0.0 Safari/540.0",
33
+ "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/9.0.600.0 Safari/534.14",
34
+ "Mozilla/5.0 (X11; U; Windows NT 6; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12",
35
+ "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.0 Safari/534.13",
36
+ "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.11 Safari/534.16",
37
+ "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20",
38
+ "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.792.0 Safari/535.1",
39
+ "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.872.0 Safari/535.2",
40
+ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7",
41
+ "Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.66 Safari/535.11",
42
+ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.45 Safari/535.19",
43
+ "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
44
+ "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
45
+ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
46
+ "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko) Chrome/24.0.1295.0 Safari/537.15",
47
+ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36",
48
+ "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1467.0 Safari/537.36",
49
+ "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36",
50
+ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1623.0 Safari/537.36",
51
+ "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36",
52
+ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36",
53
+ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.38 Safari/537.36",
54
+ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36",
55
+ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
56
+ "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36",
57
+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
58
+ "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 GTB5",
59
+ "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; ko; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2",
60
+ "Mozilla/5.0 (X11; U; SunOS sun4u; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5",
61
+ "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.12) Gecko/20080214 Firefox/2.0.0.12",
62
+ "Mozilla/5.0 (Windows; U; Windows NT 5.1; cs; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8",
63
+ "Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.8.0.5) Gecko/20060819 Firefox/1.5.0.5",
64
+ "Mozilla/5.0 (Windows; U; Windows NT 5.0; es-ES; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3",
65
+ "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5",
66
+ "Mozilla/5.0 (Windows; Windows NT 6.1; rv:2.0b2) Gecko/20100720 Firefox/4.0b2",
67
+ "Mozilla/5.0 (X11; Linux x86_64; rv:2.0b4) Gecko/20100818 Firefox/4.0b4",
68
+ "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2) Gecko/20100308 Ubuntu/10.04 (lucid) Firefox/3.6 GTB7.1",
69
+ "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b7) Gecko/20101111 Firefox/4.0b7",
70
+ "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b8pre) Gecko/20101114 Firefox/4.0b8pre",
71
+ "Mozilla/5.0 (X11; Linux x86_64; rv:2.0b9pre) Gecko/20110111 Firefox/4.0b9pre",
72
+ "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b9pre) Gecko/20101228 Firefox/4.0b9pre",
73
+ "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.2a1pre) Gecko/20110324 Firefox/4.2a1pre",
74
+ "Mozilla/5.0 (X11; U; Linux amd64; rv:5.0) Gecko/20100101 Firefox/5.0 (Debian)",
75
+ "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a2) Gecko/20110613 Firefox/6.0a2",
76
+ "Mozilla/5.0 (X11; Linux i686 on x86_64; rv:12.0) Gecko/20100101 Firefox/12.0",
77
+ "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2",
78
+ "Mozilla/5.0 (X11; Ubuntu; Linux armv7l; rv:17.0) Gecko/20100101 Firefox/17.0",
79
+ "Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20130328 Firefox/21.0",
80
+ "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:22.0) Gecko/20130328 Firefox/22.0",
81
+ "Mozilla/5.0 (Windows NT 5.1; rv:25.0) Gecko/20100101 Firefox/25.0",
82
+ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:25.0) Gecko/20100101 Firefox/25.0",
83
+ "Mozilla/5.0 (Windows NT 6.1; rv:28.0) Gecko/20100101 Firefox/28.0",
84
+ "Mozilla/5.0 (X11; Linux i686; rv:30.0) Gecko/20100101 Firefox/30.0",
85
+ "Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0",
86
+ "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0",
87
+ "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0",
88
+ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:58.0) Gecko/20100101 Firefox/58.0"
89
+ ].freeze
90
+
91
+ Aitch.configure do |config|
92
+ # config.logger = Logger.new($stdout)
93
+ end
94
+
95
+ module Enumerable
96
+ def count_by
97
+ return to_enum(:count_by) unless block_given?
98
+
99
+ each_with_object(Hash.new(0)) do |item, buffer|
100
+ buffer[yield(item)] += 1
101
+ end
102
+ end
103
+ end
104
+
105
+ def random_timeout
106
+ (1..10).map {|i| (1.0 / i).round(2) }.sample
107
+ end
108
+
109
+ def refresh_list(url:, path:, verb: :get, params: nil, &block)
110
+ response = http_request(verb, url, params)
111
+ items = block.call(response)
112
+ append_to_file(path, items)
113
+ rescue StandardError => error
114
+ puts "=> [ERROR] Unable to refresh list: #{url} (#{error.class}: #{error.message})"
115
+ []
116
+ end
117
+
118
+ def append_to_file(path, new_list)
119
+ list = load_file(path) || []
120
+ list += new_list
121
+ save_file(path, list)
122
+ list
123
+ end
124
+
125
+ def save_file(path, list)
126
+ list = list.map do |item|
127
+ if item.include?("@")
128
+ mailbox, domain = item.split("@")
129
+ else
130
+ domain = item
131
+ end
132
+
133
+ domain = domain
134
+ .squish
135
+ .gsub(/\s+/, "")
136
+ .downcase
137
+
138
+ [mailbox, SimpleIDN.to_ascii(domain)].compact.join("@")
139
+ end
140
+
141
+ list = list.uniq.sort
142
+
143
+ file_path = "#{__dir__}/../data/#{path}"
144
+
145
+ FileUtils.mkdir_p(File.dirname(file_path))
146
+
147
+ File.open(file_path, "w") do |file|
148
+ file << if path.end_with?(".json")
149
+ JSON.pretty_generate(list)
150
+ else
151
+ list.join("\n")
152
+ end
153
+ end
154
+
155
+ list
156
+ end
157
+
158
+ def load_file(path, default = [])
159
+ path = File.expand_path("#{__dir__}/../data/#{path}")
160
+
161
+ return default unless File.file?(path)
162
+
163
+ if path.end_with?(".json")
164
+ JSON.parse(File.read(path))
165
+ else
166
+ File.read(path).lines.map(&:chomp)
167
+ end
168
+ rescue StandardError => error
169
+ puts "=> [ERROR] Unable to load #{path} (#{error.class}: #{error.message})"
170
+ default
171
+ end
172
+
173
+ def http_request(verb, url, params = nil)
174
+ referer = url[%r{^(https?://[^/]+/)}, 1]
175
+
176
+ headers = {
177
+ "user-agent" => USER_AGENT.sample,
178
+ "X-Requested-With" => "XMLHttpRequest",
179
+ "Referer" => referer,
180
+ "Origin" => referer
181
+ }
182
+
183
+ Aitch.public_send(verb, url: url, headers: headers, params: params, options: {expect: 200})
184
+ end
185
+
186
+ def root_domains(domains)
187
+ processed_domains = domains.map do |domain|
188
+ RootDomain.call(domain)
189
+ end
190
+
191
+ processed_domains.reject(&:nil?)
192
+ end
193
+
194
+ def thread(&block)
195
+ Thread.new do
196
+ block.call
197
+ rescue StandardError => error
198
+ puts "=> [ERROR] Thread has errored; #{error.class}: #{error.message}"
199
+ end
200
+ end
201
+
202
+ def load_github_url(url)
203
+ puts "=> Fetching #{url}"
204
+
205
+ basename = URI.parse(url).path[%r{/([^/]+/[^/]+)}, 1].tr("/", "_").tr("-", "_")
206
+ path = "disposable/#{basename}.txt"
207
+ domains = load_file(path)
208
+
209
+ ext = File.extname(url)
210
+
211
+ domains += case ext
212
+ when ".json"
213
+ JSON.parse(http_request(:get, url).body)
214
+ when ".txt"
215
+ http_request(:get, url).body.lines.map(&:chomp)
216
+ else
217
+ raise "Unknown extension"
218
+ end
219
+
220
+ append_to_file(path, domains)
221
+ domains
222
+ rescue StandardError => error
223
+ puts "=> Unable to load #{url}; #{error.class}: #{error.message}"
224
+ []
225
+ end
226
+
227
+ def normalize_list(list)
228
+ list.flatten.map(&:chomp).compact.reject(&:empty?)
229
+ end