enricher 0.0.4 → 0.0.7

Sign up to get free protection for your applications and to get access to all the features.
Files changed (15) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +119 -10
  3. data/Rakefile +9 -1
  4. data/enricher.gemspec +1 -0
  5. data/lib/enricher.rb +26 -8
  6. data/lib/enricher/bgpranking.rb +15 -17
  7. data/lib/enricher/cdn.rb +112 -0
  8. data/lib/enricher/encoder.rb +84 -55
  9. data/lib/enricher/exceptions.rb +3 -3
  10. data/lib/enricher/ipvoid.rb +78 -0
  11. data/lib/enricher/resolver.rb +30 -0
  12. data/lib/enricher/version.rb +2 -2
  13. data/lib/enricher/virustotal.rb +51 -0
  14. data/lib/vash.rb +3 -1
  15. metadata +33 -15
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 7f87c647a272e4dd0b39b45b5e3691fdf3f8fa0e
4
- data.tar.gz: 6fc0c075c318729355ee93f56c9982714d39bfad
3
+ metadata.gz: f3a26447430c0aabed962d287dd55ecb300c4b55
4
+ data.tar.gz: 8afd50139c78292f1cdd4931c966a8f0414675cf
5
5
  SHA512:
6
- metadata.gz: d883c35af6c5d1542aaade6b622f82b5c3044a78633fba1abc250a00c4851c53b2e6aef4c895b84fca7044657c1e7c16dbac34843d17b11bea9ebdb9c685510a
7
- data.tar.gz: d88bca4c18a7c68e74490492a8763d5e5b83f300c257d6cf421e289152ed98d161a28e243bff86d2e2084cbca3c63f1df4afbdb4443f150b14e3624a886d6f9b
6
+ metadata.gz: 4d7c1bca11cc0e5cec806a22fa4fb878492673c24c479b1b3d0a786c013b09258dec97b355ed4597bca3a1820029a95f388c0b41a01a45669168be13545e8bfb
7
+ data.tar.gz: 378842b2949ad62ea3f0718600bfccbe0581a3339408e9287e388afb7ebd412d76601c33f72b9ea40677bec02c143df9e70629b20a8910bb50b2720badd85de4
data/README.md CHANGED
@@ -2,33 +2,142 @@
2
2
 
3
3
  IPv4 Data Enricher
4
4
 
5
+ [![Gem Version](https://badge.fury.io/rb/enricher.png)](http://badge.fury.io/rb/enricher)
6
+
5
7
  ### Static Calculators:
6
8
 
7
- Calculate ASN, CC3, Bogon, and Lat Long.
9
+ Calculate Maxmind ASN, CC3, Bogon inclusion, and geodata such as Lat Long.
10
+
11
+ CDN Hostname check regex lookup.
8
12
 
9
13
  ### Online Calculators:
10
14
 
11
- IPv4 reputation with VOIDIP
12
- IPv4 reputation with VirusTotal
15
+ BGP Ranking with http://bgpranking.circl.lu/ ASN Calculator
16
+
17
+ Full Bogon Checking with Team Cymru List
18
+
19
+ Reverse DNS Lookups from (L3 DNS Servers 4.2.2.2/3/4)
20
+
21
+
22
+ ### Usage
23
+
24
+ ```
25
+ >> require 'enricher'
26
+ => true
27
+ ```
28
+
29
+ #### For Pure Offline Meta Data Enhancement
30
+ ```
31
+ >> a = Enricher::Encoder.encode('10.48.185.173')
32
+ => {:ip=>170965421, :asn=>"--", :asn_rank=>"0.0", :geoip=>"--", :bogon=>true}
33
+
34
+ >> a = Enricher::Encoder.encode('108.48.185.173')
35
+ => {:ip=>1815132589, :asn=>"AS701", :asn_rank=>"0.000011", :geoip=>"USA", :bogon=>false}
36
+ ```
37
+
38
+ #### For Online Meta Data Enrichment
39
+ ```
40
+ 2.1.2 :006 > a = Enricher::Encoder.encode_online('96.3.8.26')
41
+ => {:ip=>1610811418, :asn=>"AS11232", :asn_rank=>"0.000048", :geoip=>"USA", :bogon=>false, :reverse=>"host-26-8-3-96.midco.net.", :cdn=>false}
42
+ 2.1.2 :007 > a = Enricher::Encoder.encode_online('96.6.113.42')
43
+ => {:ip=>1611034922, :asn=>"AS20940", :asn_rank=>"0.000411", :geoip=>"USA", :bogon=>false, :reverse=>"a96-6-113-42.deploy.akamaitechnologies.com.", :cdn=>true}
44
+ 2.1.2 :008 > a = Enricher::Encoder.encode_online('119.27.76.185')
45
+ => {:ip=>1998277817, :asn=>"AS4837", :asn_rank=>"0.000082", :geoip=>"CHN", :bogon=>false, :reverse=>"", :cdn=>false}
46
+ 2.1.2 :009 > a = Enricher::Encoder.encode_online('199.27.76.185')
47
+ => {:ip=>3340455097, :asn=>"AS54113", :asn_rank=>"0.000526", :geoip=>"USA", :bogon=>false, :reverse=>"", :cdn=>false}
48
+ 2.1.2 :010 > a = Enricher::Encoder.encode_online('54.239.195.35')
49
+ => {:ip=>921682723, :asn=>"AS16509", :asn_rank=>"0.000293", :geoip=>"USA", :bogon=>false, :reverse=>"server-54-239-195-35.nrt12.r.cloudfront.net.", :cdn=>true}
50
+ 2.1.2 :011 > a = Enricher::Encoder.encode_online('54.239.195.149')
51
+ => {:ip=>921682837, :asn=>"AS16509", :asn_rank=>"0.000293", :geoip=>"USA", :bogon=>false, :reverse=>"server-54-239-195-149.nrt12.r.cloudfront.net.", :cdn=>true}
52
+ 2.1.2 :012 > a = Enricher::Encoder.encode_online('72.21.91.8')
53
+ => {:ip=>1209359112, :asn=>"AS15133", :asn_rank=>"0.000312", :geoip=>"USA", :bogon=>false, :reverse=>"", :cdn=>false}
54
+ 2.1.2 :013 > a = Enricher::Encoder.encode_online('173.194.121.44')
55
+ => {:ip=>2915203372, :asn=>"AS15169", :asn_rank=>"0.000204", :geoip=>"USA", :bogon=>false, :reverse=>"iad23s26-in-f12.1e100.net.", :cdn=>true}
56
+ 2.1.2 :014 > a = Enricher::Encoder.encode_online('205.234.175.175')
57
+ => {:ip=>3454709679, :asn=>"AS30081", :asn_rank=>"0.000781", :geoip=>"USA", :bogon=>false, :reverse=>"vip1.G-anycast1.cachefly.net.", :cdn=>true}
58
+ ```
59
+
60
+ ### Use of Volatile Hash DB
61
+
62
+ BGP Scores are stored for 12 hours in a volatile hash cache.
63
+
64
+ ```
65
+ 2.1.2 :015 > Enricher::BGPRanking.cache
66
+ => {:asn12542=>"0.000010", :asn11232=>"0.000048", :asn20940=>"0.000411", :asn4837=>"0.000082", :asn54113=>"0.000526", :asn16509=>"0.000293", :asn15133=>"0.000312", :asn15169=>"0.000204", :asn30081=>"0.000781"}
67
+ ```
68
+
69
+ ### TODO
13
70
 
14
- BGP Ranking with Cyrmu ASN Calculator
71
+ * IPv4 reputation with VOIDIP
72
+ * IPv4 reputation with VirusTotal (uirusu)
15
73
 
16
74
  ## Requirements
17
75
 
18
- Intenet connectivity for Online Calculators.
76
+ Intenet connectivity is required *only* for Online Meta Data Enhancement.
77
+
78
+ ### BootStrapping GeoIP
79
+
80
+ There is a rake command include to bootstrap the fetching of geoip data if you are a sudo enabled linux user.
81
+
82
+ ```
83
+ (/home/shadowbq/.rvm/gems/ruby-2.1.2/gems/enricher-0.0.7)$ rake fetch_geoip_data
84
+ --2014-12-31 11:32:28-- http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
85
+ Resolving geolite.maxmind.com (geolite.maxmind.com)... 141.101.115.190, 141.101.114.190, 2400:cb00:2048:1::8d65:72be, ...
86
+ Connecting to geolite.maxmind.com (geolite.maxmind.com)|141.101.115.190|:80... connected.
87
+ HTTP request sent, awaiting response... 200 OK
88
+ Length: 428181 (418K) [application/octet-stream]
89
+ Saving to: ‘/usr/local/lib/share/enricher/GeoIP.dat.gz’
90
+
91
+ 100%[=====================================================================================================================================================================================================================================>] 428,181 --.-K/s in 0.08s
92
+
93
+ 2014-12-31 11:32:28 (5.22 MB/s) - ‘/usr/local/lib/share/enricher/GeoIP.dat.gz’ saved [428181/428181]
94
+
95
+ --2014-12-31 11:32:28-- http://geolite.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
96
+ Resolving geolite.maxmind.com (geolite.maxmind.com)... 141.101.114.190, 141.101.115.190, 2400:cb00:2048:1::8d65:73be, ...
97
+ Connecting to geolite.maxmind.com (geolite.maxmind.com)|141.101.114.190|:80... connected.
98
+ HTTP request sent, awaiting response... 200 OK
99
+ Length: 2056964 (2.0M) [application/octet-stream]
100
+ Saving to: ‘/usr/local/lib/share/enricher/GeoIPASNum.dat.gz’
101
+
102
+ 100%[=====================================================================================================================================================================================================================================>] 2,056,964 1.38MB/s in 1.4s
103
+
104
+ 2014-12-31 11:32:30 (1.38 MB/s) - ‘/usr/local/lib/share/enricher/GeoIPASNum.dat.gz’ saved [2056964/2056964]
105
+
106
+ --2014-12-31 11:32:30-- http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
107
+ Resolving geolite.maxmind.com (geolite.maxmind.com)... 141.101.115.190, 141.101.114.190, 2400:cb00:2048:1::8d65:72be, ...
108
+ Connecting to geolite.maxmind.com (geolite.maxmind.com)|141.101.115.190|:80... connected.
109
+ HTTP request sent, awaiting response... 200 OK
110
+ Length: 11896864 (11M) [application/octet-stream]
111
+ Saving to: ‘/usr/local/lib/share/enricher/GeoLiteCity.dat.gz’
112
+
113
+ 100%[=====================================================================================================================================================================================================================================>] 11,896,864 5.10MB/s in 2.2s
114
+
115
+ 2014-12-31 11:32:32 (5.10 MB/s) - ‘/usr/local/lib/share/enricher/GeoLiteCity.dat.gz’ saved [11896864/11896864]
116
+
117
+ >$ ls -la /usr/local/lib/share/enricher/
118
+ total 22664
119
+ drwxr-xr-x 2 root root 4096 Dec 31 11:32 .
120
+ drwxr-xr-x 3 root root 4096 Dec 31 11:23 ..
121
+ -rw-r--r-- 1 root root 748606 Dec 2 16:43 GeoIP.dat
122
+ -rw-r--r-- 1 root root 3766172 Dec 12 17:54 GeoIPASNum.dat
123
+ -rw-r--r-- 1 root root 18678957 Dec 2 16:57 GeoLiteCity.dat
124
+
125
+ ```
126
+
127
+ (Manual Methods)
19
128
 
20
- Maxmind dat file location requirement: `/usr/local/lib/share/enricher`
129
+ Maxmind(R) dat file location requirement: `/usr/local/lib/share/enricher`
21
130
 
22
131
  You need to download and install each of the free GeoLite country, city or ASN databases, or a subscription database version.
23
132
 
24
133
  The last known download locations for the GeoLite database versions are:
25
134
 
26
- <geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz>
27
- <geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz>
28
- <geolite.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz>
135
+ * http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
136
+ * http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
137
+ * http://geolite.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
29
138
 
30
139
 
31
- ## Automating the update of Static Content (via crontrab)
140
+ ### Automating the update of Static Content (via crontrab)
32
141
 
33
142
  We can add a cron-job to automate the monthly process of updating the GeoIP database:
34
143
 
data/Rakefile CHANGED
@@ -8,4 +8,12 @@ Rake::TestTask.new do |test|
8
8
  test.libs << "test"
9
9
  test.test_files = Dir[ "test/test_*.rb" ]
10
10
  test.verbose = true
11
- end
11
+ end
12
+
13
+ desc 'Fetch the GeoIP Data sets'
14
+ task :fetch_geoip_data do
15
+ `sudo mkdir -p /usr/local/lib/share/enricher`
16
+ `sudo sh -c '/usr/bin/wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz -O /usr/local/lib/share/enricher/GeoIP.dat.gz; /bin/gunzip -f /usr/local/lib/share/enricher/GeoIP.dat.gz'`
17
+ `sudo sh -c '/usr/bin/wget http://geolite.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz -O /usr/local/lib/share/enricher/GeoIPASNum.dat.gz; /bin/gunzip -f /usr/local/lib/share/enricher/GeoIPASNum.dat.gz'`
18
+ `sudo sh -c '/usr/bin/wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz -O /usr/local/lib/share/enricher/GeoLiteCity.dat.gz; /bin/gunzip -f /usr/local/lib/share/enricher/GeoLiteCity.dat.gz'`
19
+ end
data/enricher.gemspec CHANGED
@@ -18,6 +18,7 @@ Gem::Specification.new do |gem|
18
18
  gem.add_development_dependency 'bundler', '~> 1.0'
19
19
  gem.add_dependency "geoip", '~> 1.2'
20
20
  gem.add_dependency "netaddr", '~> 1.5'
21
+ gem.add_dependency "net-dns", '~> 0.8'
21
22
  gem.add_dependency "rest-client"
22
23
  gem.add_dependency "json"
23
24
 
data/lib/enricher.rb CHANGED
@@ -13,7 +13,8 @@ require 'json'
13
13
  require 'geoip'
14
14
  require 'netaddr'
15
15
  require 'rest-client'
16
-
16
+ require 'net/dns'
17
+ #require 'uirusu'
17
18
 
18
19
  # Internal
19
20
  module Enricher
@@ -24,14 +25,24 @@ module Enricher
24
25
  require 'enricher/exceptions'
25
26
  require 'enricher/bogon'
26
27
  require 'enricher/bgpranking'
28
+ require 'enricher/cdn'
29
+ require 'enricher/resolver'
27
30
  require 'enricher/encoder'
28
31
 
29
32
  DEBUG=false
30
33
  LOGGING=false
31
-
34
+ COMMON_DATA_PATHS=[
35
+ '/usr/local/lib/share/enricher',
36
+ '/usr/local/share/enricher',
37
+ '/usr/local/lib/enricher',
38
+ '/usr/local/etc/enricher',
39
+ '/etc/enricher',
40
+ '/var/db/enricher'
41
+ ]
42
+
32
43
  #Setup Paths
33
44
  LIB_PATH = File.expand_path("../", __FILE__)
34
- CONFIG_PATH = File.expand_path("../../db", __FILE__)
45
+ #CONFIG_PATH = File.expand_path("../../db", __FILE__)
35
46
 
36
47
  if Enricher::DEBUG
37
48
  Enricher::DATA_PATH = File.expand_path("../../data", __FILE__)
@@ -39,11 +50,18 @@ module Enricher
39
50
  Enricher::LOG_PATH = File.expand_path("../../log", __FILE__)
40
51
  logfile = "#{Enricher::LOG_PATH}/enricher.log"
41
52
  end
42
- else
43
- Enricher::DATA_PATH = File.path("/usr/local/lib/share/enricher")
44
- if Enricher::LOGGING
45
- logfile = Tempfile.new('enricher.log')
46
- Enricher::LOG_PATH = File.dirname(logfile.path)
53
+ else
54
+ COMMON_DATA_PATHS.each do |dirname|
55
+ if File.exists?(dirname) && File.directory?(dirname)
56
+ Enricher::DATA_PATH = File.path(dirname)
57
+ if Enricher::LOGGING
58
+ logfile = Tempfile.new('enricher.log')
59
+ Enricher::LOG_PATH = File.dirname(logfile.path)
60
+ end
61
+ break
62
+ else
63
+ raise EnricherPathMissing, "Enricher data path not found in Common Data Paths. (i.e /usr/local/lib/share/enricher) See README.md"
64
+ end
47
65
  end
48
66
  end
49
67
 
data/lib/enricher/bgpranking.rb CHANGED
@@ -9,33 +9,31 @@ module Enricher
9
9
  # >> r = RestClient.post "http://bgpranking.circl.lu/json", { 'method' => 'cached_daily_rank', 'asn' => 198540, 'date' => '2014-02-24' }.to_json, :content_type => :json, :accept => :json
10
10
  # => "[198540, "ELAN-AS Przedsiebiorstwo Uslug Specjalistycznych ELAN mgr inz. Andrzej Niechcial", "2014-02-24", "global", 1.09609375]"
11
11
  # >> a = JSON.parse(r)
12
-
13
12
  class BGPRanking
14
13
 
15
14
  BGP_RANK_URL = "http://bgpranking.circl.lu/json"
16
15
 
17
- def self.rank?(addr)
18
- asn = addr.strip[/[0-9]+/]
19
- if asn =~ /[0-9]+/
20
- @@cache ||= Vash.new
21
- # Voliate Cache store for 43200 (12hr)
22
- @@cache["asn#{asn}".to_sym] ||= self.onlinerank?(asn)
23
- else
24
- return "0.0"
16
+ def self.rank?(addr)
17
+ asn = addr.strip[/[0-9]+/]
18
+ if asn =~ /[0-9]+/
19
+ @@cache ||= Vash.new
20
+ # Voliate Cache store for 43200 (12hr)
21
+ @@cache["asn#{asn}".to_sym] ||= self.onlinerank?(asn)
22
+ else
23
+ return "0.0"
25
24
  end
26
- end
25
+ end
27
26
 
28
27
  def self.cache
29
- @@cache
28
+ @@cache
30
29
  end
31
30
 
32
31
  private
33
32
 
34
- def self.onlinerank?(addr)
35
- resp = RestClient.post BGP_RANK_URL, { 'method' => 'cached_daily_rank', 'asn' => addr, 'date' => Date.strptime((Date.today - 1).to_s, '%Y-%m-%d').to_s }.to_json, :content_type => :json, :accept => :json
36
- return "%.6f" % JSON.parse(resp)[4]
37
- end
33
+ def self.onlinerank?(addr)
34
+ resp = RestClient.post BGP_RANK_URL, { 'method' => 'cached_daily_rank', 'asn' => addr, 'date' => Date.strptime((Date.today - 1).to_s, '%Y-%m-%d').to_s }.to_json, :content_type => :json, :accept => :json
35
+ return "%.6f" % JSON.parse(resp)[4]
36
+ end
38
37
 
39
38
  end
40
-
41
- end
39
+ end
data/lib/enricher/cdn.rb ADDED
@@ -0,0 +1,112 @@
1
+ module Enricher
2
+ #
3
+ # Bogons ipv4 allow for both static simple checks and for dynamic full Bogon list checks provided by Team Cymru.
4
+ #
5
+ # CDN List provided by - (23 Dec 2014) https://github.com/WPO-Foundation/webpagetest/blob/master/agent/wpthook/cdn.h
6
+ #
7
+ class CDN
8
+
9
+ CDN_PROVIDERS = [
10
+ [".akamai.net", "Akamai"],
11
+ [".akamaiedge.net", "Akamai"],
12
+ [".akamaihd.net", "Akamai"],
13
+ [".edgesuite.net", "Akamai"],
14
+ [".edgekey.net", "Akamai"],
15
+ [".srip.ne", "Akamai"],
16
+ [".akamaitechnologies.com", "Akamai"],
17
+ [".akamaitechnologies.fr", "Akamai"],
18
+ [".llnwd.net", "Limelight"],
19
+ ["edgecastcdn.net", "Edgecast"],
20
+ [".systemcdn.net", "Edgecast"],
21
+ [".transactcdn.net", "Edgecast"],
22
+ [".v1cdn.net", "Edgecast"],
23
+ [".v2cdn.net", "Edgecast"],
24
+ [".v3cdn.net", "Edgecast"],
25
+ [".v4cdn.net", "Edgecast"],
26
+ [".v5cdn.net", "Edgecast"],
27
+ ["hwcdn.net", "Highwinds"],
28
+ [".simplecdn.net", "Simple CDN"],
29
+ [".instacontent.net", "Mirror Image"],
30
+ [".footprint.net", "Level 3"],
31
+ [".ay1.b.yahoo.com", "Yahoo"],
32
+ [".yimg.", "Yahoo"],
33
+ [".yahooapis.com", "Yahoo"],
34
+ [".google.", "Google"],
35
+ ["googlesyndication.", "Google"],
36
+ ["youtube.", "Google"],
37
+ [".googleusercontent.com", "Google"],
38
+ ["googlehosted.com", "Google"],
39
+ [".gstatic.com", "Google"],
40
+ [".insnw.net", "Instart Logic"],
41
+ [".inscname.net", "Instart Logic"],
42
+ [".internapcdn.net", "Internap"],
43
+ [".cloudfront.net", "Amazon CloudFront"],
44
+ [".netdna-cdn.com", "NetDNA"],
45
+ [".netdna-ssl.com", "NetDNA"],
46
+ [".netdna.com", "NetDNA"],
47
+ [".cotcdn.net", "Cotendo CDN"],
48
+ [".cachefly.net", "Cachefly"],
49
+ ["bo.lt", "BO.LT"],
50
+ [".cloudflare.com", "Cloudflare"],
51
+ [".afxcdn.net", "afxcdn.net"],
52
+ [".lxdns.com", "ChinaNetCenter"],
53
+ [".att-dsa.net", "AT&T"],
54
+ [".vo.msecnd.net", "Windows Azure"],
55
+ [".voxcdn.net", "VoxCDN"],
56
+ [".bluehatnetwork.com", "Blue Hat Network"],
57
+ [".swiftcdn1.com", "SwiftCDN"],
58
+ [".cdngc.net", "CDNetworks"],
59
+ [".gccdn.net", "CDNetworks"],
60
+ [".panthercdn.com", "CDNetworks"],
61
+ [".fastly.net", "Fastly"],
62
+ [".nocookie.net", "Fastly"],
63
+ [".gslb.taobao.com", "Taobao"],
64
+ [".gslb.tbcache.com", "Alimama"],
65
+ [".mirror-image.net", "Mirror Image"],
66
+ [".yottaa.net", "Yottaa"],
67
+ [".cubecdn.net", "cubeCDN"],
68
+ [".r.cdn77.net", "CDN77"],
69
+ [".incapdns.net", "Incapsula"],
70
+ [".bitgravity.com", "BitGravity"],
71
+ [".r.worldcdn.net", "OnApp"],
72
+ [".r.worldssl.net", "OnApp"],
73
+ ["tbcdn.cn", "Taobao"],
74
+ [".taobaocdn.com", "Taobao"],
75
+ [".ngenix.net", "NGENIX"],
76
+ [".pagerain.net", "PageRain"],
77
+ [".ccgslb.com", "ChinaCache"],
78
+ ["cdn.sfr.net", "SFR"],
79
+ [".azioncdn.net", "Azion"],
80
+ [".azioncdn.com", "Azion"],
81
+ [".azion.net", "Azion"],
82
+ [".cdncloud.net.au", "MediaCloud"],
83
+ [".rncdn1.com", "Reflected Networks"],
84
+ [".cdnsun.net", "CDNsun"],
85
+ [".mncdn.com", "Medianova"],
86
+ [".mncdn.net", "Medianova"],
87
+ [".mncdn.org", "Medianova"],
88
+ ["cdn.jsdelivr.net", "jsDelivr"],
89
+ [".nyiftw.net", "NYI FTW"],
90
+ [".nyiftw.com", "NYI FTW"],
91
+ [".resrc.it", "ReSRC.it"],
92
+ [".zenedge.net", "Zenedge"],
93
+ [".lswcdn.net", "LeaseWeb CDN"],
94
+ [".revcn.net", "Rev Software"],
95
+ [".revdn.net", "Rev Software"],
96
+ [".1e100.net", "Google"]
97
+ ]
98
+
99
+ # Expects a hostname string as a variable
100
+ def self.contains?(hostname = "")
101
+ if hostname != "" then
102
+ CDN_PROVIDERS.each { |cdn_entry|
103
+ if hostname.match(cdn_entry[0]) then
104
+ return true
105
+ end
106
+ }
107
+ end
108
+ return false
109
+ end
110
+
111
+ end
112
+ end
data/lib/enricher/encoder.rb CHANGED
@@ -1,70 +1,99 @@
1
1
  module Enricher
2
-
2
+
3
3
  class Encoder
4
-
5
- def self.encode(ip)
6
-
7
- @@geoASN ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIPASNum.dat")
8
- @@geoCoder ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIP.dat")
9
- @@geoCoderCity ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoLiteCity.dat")
4
+
5
+ ## Class Methods for the Encoder..
6
+ class << self
7
+
8
+
9
+ def encode(ip)
10
+
11
+ @@geoASN ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIPASNum.dat")
12
+ @@geoCoder ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIP.dat")
13
+ @@geoCoderCity ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoLiteCity.dat")
14
+ @@res ||= Enricher::Resolver.new
15
+
16
+ @@bogon_type ||= self.bogon_type
17
+ @@bogon ||= Bogon.new(@@bogon_type)
18
+
19
+ asn = @@geoASN.asn(ip).number rescue "--"
20
+
21
+ {:ip => IPAddr.new(ip).to_i, :asn => asn, :asn_rank => Enricher::BGPRanking.rank?(asn), :geoip => @@geoCoder.country(ip).country_code3, :bogon => @@bogon.contains?(ip)}
22
+ end
10
23
 
11
- @@bogon_type ||= self.bogon_type
12
- @@bogon ||= Bogon.new(@@bogon_type)
24
+ def encode_online(ip)
25
+ @@geoASN ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIPASNum.dat")
26
+ @@geoCoder ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIP.dat")
27
+ @@geoCoderCity ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoLiteCity.dat")
28
+ @@res ||= Enricher::Resolver.new
29
+
30
+ @@bogon_type ||= self.bogon_type(:live)
31
+ @@bogon ||= Bogon.new(@@bogon_type)
32
+
33
+ asn = @@geoASN.asn(ip).number rescue "--"
34
+ reverse_hostname = self.reverse(ip) rescue ""
35
+
36
+ {:ip => IPAddr.new(ip).to_i, :asn => asn, :asn_rank => Enricher::BGPRanking.rank?(asn), :geoip => @@geoCoder.country(ip).country_code3, :bogon => @@bogon.contains?(ip), :reverse => reverse_hostname, :cdn => self.cdn?(reverse_hostname)}
37
+ end
38
+
39
+ def aton(a)
40
+ IPAddr.new(a).to_i
41
+ end
42
+
43
+ def ntoa(a)
44
+ IPAddr.new(a, Socket::AF_INET).to_s
45
+ end
46
+
47
+ def rank?(asn)
48
+ Enricher::BGPRanking.rank?(asn)
49
+ end
50
+
51
+ def bogon?(ip)
52
+ @@bogon_type ||= self.bogon_type
53
+ @@bogon ||= Bogon.new(@@bogon_type)
54
+ return @@bogon.contains?(ip)
55
+ end
13
56
 
14
- asn = @@geoASN.asn(ip).number rescue "--"
57
+ def bogon_type(bogon_sym=:ipv4)
58
+ bogon_sym
59
+ end
60
+
61
+ def asn(ip)
62
+ @@geoASN ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIPASNum.dat")
63
+ return @@geoASN.asn(ip).number rescue "--"
64
+ end
15
65
 
16
- {:ip => IPAddr.new(ip).to_i, :asn => asn, :asn_rank => Enricher::BGPRanking.rank?(asn), :geoip => @@geoCoder.country(ip).country_code3, :bogon => @@bogon.contains?(ip)}
17
- end
18
-
19
- def self.aton(a)
20
- IPAddr.new(a).to_i
21
- end
66
+ def asn_company(ip)
67
+ @@geoASN ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIPASNum.dat")
68
+ return @@geoASN.asn(ip).asn rescue "--"
69
+ end
22
70
 
23
- def self.ntoa(a)
24
- IPAddr.new(a, Socket::AF_INET).to_s
25
- end
71
+ def cc3(ip)
72
+ @@geoCoder ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIP.dat")
73
+ return @@geoCoder.country(ip).country_code3
74
+ end
26
75
 
27
- def self.rank?(asn)
28
- Enricher::BGPRanking.rank?(asn)
29
- end
76
+ def latitude(ip)
77
+ @@geoCoderCity ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIP.dat")
78
+ return @@geoCoderCity.city(ip).latitude
79
+ end
30
80
 
31
- def self.bogon?(ip)
32
- @@bogon_type ||= self.bogon_type
33
- @@bogon ||= Bogon.new(@@bogon_type)
34
- return @@bogon.contains?(ip)
35
- end
36
-
37
- def self.bogon_type(bogon_sym=:ipv4)
38
- bogon_sym
39
- end
81
+ def longitude(ip)
82
+ @@geoCoderCity ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIP.dat")
83
+ return @@geoCoderCity.city(ip).longitude
84
+ end
40
85
 
41
- def self.asn(ip)
42
- @@geoASN ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIPASNum.dat")
43
- return @@geoASN.asn(ip).number rescue "--"
44
- end
45
-
46
- def self.asn_company(ip)
47
- @@geoASN ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIPASNum.dat")
48
- return @@geoASN.asn(ip).asn rescue "--"
49
- end
50
-
51
- def self.cc3(ip)
52
- @@geoCoder ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIP.dat")
53
- return @@geoCoder.country(ip).country_code3
54
- end
86
+ def reverse(ip)
87
+ @@res ||= Enricher::Resolver.new
88
+ return @@res.reverse?(ip)
89
+ end
55
90
 
56
- def self.latitude(ip)
57
- @@geoCoderCity ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIP.dat")
58
- return @@geoCoderCity.city(ip).latitude
59
- end
91
+ def cdn?(hostname)
92
+ return Enricher::CDN.contains?(hostname)
93
+ end
60
94
 
61
- def self.longitude(ip)
62
- @@geoCoderCity ||= GeoIP.new("#{Enricher::DATA_PATH}/GeoIP.dat")
63
- return @@geoCoderCity.city(ip).longitude
64
95
  end
65
96
 
66
-
67
97
  end
68
98
 
69
- end
70
-
99
+ end
data/lib/enricher/exceptions.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  module Enricher
2
-
3
2
  class BogonSetUndefined < StandardError; end
4
-
5
- end
3
+ class EnricherPathMissing < StandardError; end
4
+ class DisabledClassIncluded < StandardError; end
5
+ end
data/lib/enricher/ipvoid.rb ADDED
@@ -0,0 +1,78 @@
1
+ module Enricher
2
+ #
3
+ # IPVOID ipv4 allow for dynamic checks against the list checks provided by IPVOID.
4
+ #
5
+ class IPVoid
6
+
7
+ DISABLED = true
8
+
9
+ def self.url_cache
10
+ @@url_cache
11
+ end
12
+
13
+ def self.hash_cache
14
+ @@hash_cache
15
+ end
16
+
17
+ def initialize(constructor = {})
18
+
19
+ raise DisabledClassIncluded if DISABLED
20
+ #First you need to include the correct require files
21
+ APT_KEY = "YOUR API KEY HERE"
22
+ @@hash_cache ||= Vash.new
23
+ @@url_cache ||= Vash.new
24
+ # Voliate Cache store for 43200 (12hr)
25
+ end
26
+
27
+ def junk
28
+
29
+ # RestClient scrape with Nokogiri.... (nokogiri requires libxml which is native which is not jruby compliant.. )
30
+
31
+ =begin
32
+ for ip in open(conf.iplist, "r"):
33
+ url = "http://www.ipvoid.com/scan/%s" % (ip)
34
+ emailBody = emailBody + "IP: "+ip
35
+ resp = requests.get(url)
36
+ string1 = unicodedata.normalize('NFKD', resp.text).encode('ascii','ignore')
37
+ r = string1.translate(string.maketrans("\n\t\r", " "))
38
+ blacklist = re.search(r'Blacklist Status</td><td><span.+>(\w.+)</span>', r)
39
+ if blacklist != None and blacklist.group(1) == "BLACKLISTED":
40
+ emailBody = emailBody + 'The IP is blacklisted! \n'
41
+ detection = re.search(r'Detection Ratio</td><td>(\d+ / \d+) \(<font', r)
42
+ emailBody = emailBody + 'Detection Ratio was %s \n' % detection.group(1)
43
+ detected_line = re.search(r'\s+<tr><td><img src="(.+)', r)
44
+ detected_sites = re.findall(r'Favicon" />(.+?)</td><td><img src=".+?" alt="Alert" title="Detected!".+?"nofollow" href="(.+?)" title', detected_line.group(1))
45
+ for site in detected_sites:
46
+ emailBody = emailBody + "List Name:" + site[0] + "Url: "+ site[1] + "\n\n"
47
+ else:
48
+ emailBody = emailBody + 'Not blacklisted...\n\n'
49
+ =end
50
+ end
51
+
52
+
53
+ def hash(hash)
54
+ #To query a hash(sha1/sha256/md5)
55
+ @@hash_cache["vt_#{hash}".to_sym] ||= Uirusu::VTFile.query_report(VT_APT_KEY, hash)
56
+ result = Uirusu::VTResult.new(hash, results)
57
+ result.to_json
58
+ end
59
+
60
+ def url(url)
61
+
62
+ # Use Base 36 for symbols
63
+ #>> "joe@momma.org".hash.to_s(36)
64
+ #=> "37zed965f04p"
65
+ #>> "http://joe@momma.org".hash.to_s(36)
66
+ #=> "vj36lppwievl"
67
+ #=> Tack on.. vt_ to url converted .hash.to_s(36)
68
+
69
+ @@url_cache["vt_#{url.hash.to_s(36)}".to_sym] ||= Uirusu::VTUrl.query_report(VT_APT_KEY, url)
70
+
71
+ result = Uirusu::VTResult.new(url, results)
72
+ result.to_json
73
+ end
74
+
75
+
76
+ end
77
+
78
+ end
data/lib/enricher/resolver.rb ADDED
@@ -0,0 +1,30 @@
1
+ module Enricher
2
+ #
3
+ # Bogons ipv4 allow for both static simple checks and for dynamic full Bogon list checks provided by Team Cymru.
4
+ #
5
+ # >> @@mybogon = Enricher::Bogon.new(:live)^C
6
+ # >> @@mybogon.contains?('205.166.22.1')
7
+ # => true
8
+ # >> @@mybogon = Enricher::Bogon.new(:ipv4)
9
+ # => #<Enricher::Bogon:0x00000002fb0368 @bogon=[0.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 192.0.0.0/24, 192.0.2.0/24, 192.168.0.0/16, 198.18.0.0/15, 198.51.100.0/24, 203.0.113.0/24, 224.0.0.0/4, 240.0.0.0/4]>
10
+ # >> @@mybogon.contains?('205.166.22.1')
11
+ # => false
12
+ #
13
+ class Resolver
14
+
15
+ def initialize(nameservers = ["4.2.2.2","4.2.2.3","4.2.2.4"])
16
+ @res = Net::DNS::Resolver.new
17
+ @res.nameservers = nameservers
18
+ end
19
+
20
+ def reverse?(ip)
21
+ begin
22
+ packet = @res.search(ip)
23
+ return packet.answer[0].ptr
24
+ rescue
25
+ return ""
26
+ end
27
+ end
28
+
29
+ end
30
+ end
data/lib/enricher/version.rb CHANGED
@@ -1,3 +1,3 @@
1
1
  module Enricher
2
- VERSION = '0.0.4'
3
- end
2
+ VERSION = '0.0.7'
3
+ end
data/lib/enricher/virustotal.rb ADDED
@@ -0,0 +1,51 @@
1
+ module Enricher
2
+ #
3
+ # Bogons ipv4 allow for both static simple checks and for dynamic full Bogon list checks provided by Team Cymru.
4
+ #
5
+ class VirusTotal
6
+
7
+ DISABLED = true
8
+
9
+ def self.url_cache
10
+ @@url_cache
11
+ end
12
+
13
+ def self.hash_cache
14
+ @@hash_cache
15
+ end
16
+
17
+ def initialize(constructor = {})
18
+
19
+ raise DisabledClassIncluded if DISABLED
20
+ #First you need to include the correct require files
21
+ APT_KEY = "YOUR API KEY HERE"
22
+ @@hash_cache ||= Vash.new
23
+ @@url_cache ||= Vash.new
24
+ # Voliate Cache store for 43200 (12hr)
25
+ end
26
+
27
+ def hash(hash)
28
+ #To query a hash(sha1/sha256/md5)
29
+ @@hash_cache["vt_#{hash}".to_sym] ||= Uirusu::VTFile.query_report(VT_APT_KEY, hash)
30
+ result = Uirusu::VTResult.new(hash, results)
31
+ result.to_json
32
+ end
33
+
34
+ def url(url)
35
+
36
+ # Use Base 36 for symbols
37
+ #>> "joe@momma.org".hash.to_s(36)
38
+ #=> "37zed965f04p"
39
+ #>> "http://joe@momma.org".hash.to_s(36)
40
+ #=> "vj36lppwievl"
41
+ #=> Tack on.. vt_ to url converted .hash.to_s(36)
42
+
43
+ @@url_cache["vt_#{url.hash.to_s(36)}".to_sym] ||= Uirusu::VTUrl.query_report(VT_APT_KEY, url)
44
+
45
+ result = Uirusu::VTResult.new(url, results)
46
+ result.to_json
47
+ end
48
+
49
+ end
50
+
51
+ end
data/lib/vash.rb CHANGED
@@ -91,7 +91,8 @@ class Vash < Hash
91
91
  self.delete key
92
92
  end
93
93
 
94
- private
94
+ private
95
+
95
96
  def expired?(key)
96
97
  Time.now.to_i > @register[key].to_i
97
98
  end
@@ -107,4 +108,5 @@ private
107
108
  def sterilize(key)
108
109
  key = sterile(key)
109
110
  end
111
+
110
112
  end
metadata CHANGED
@@ -1,83 +1,97 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: enricher
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.4
4
+ version: 0.0.7
5
5
  platform: ruby
6
6
  authors:
7
7
  - shadowbq
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2014-02-25 00:00:00.000000000 Z
11
+ date: 2014-12-31 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
15
15
  requirement: !ruby/object:Gem::Requirement
16
16
  requirements:
17
- - - ~>
17
+ - - "~>"
18
18
  - !ruby/object:Gem::Version
19
19
  version: '1.0'
20
20
  type: :development
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
- - - ~>
24
+ - - "~>"
25
25
  - !ruby/object:Gem::Version
26
26
  version: '1.0'
27
27
  - !ruby/object:Gem::Dependency
28
28
  name: geoip
29
29
  requirement: !ruby/object:Gem::Requirement
30
30
  requirements:
31
- - - ~>
31
+ - - "~>"
32
32
  - !ruby/object:Gem::Version
33
33
  version: '1.2'
34
34
  type: :runtime
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
- - - ~>
38
+ - - "~>"
39
39
  - !ruby/object:Gem::Version
40
40
  version: '1.2'
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: netaddr
43
43
  requirement: !ruby/object:Gem::Requirement
44
44
  requirements:
45
- - - ~>
45
+ - - "~>"
46
46
  - !ruby/object:Gem::Version
47
47
  version: '1.5'
48
48
  type: :runtime
49
49
  prerelease: false
50
50
  version_requirements: !ruby/object:Gem::Requirement
51
51
  requirements:
52
- - - ~>
52
+ - - "~>"
53
53
  - !ruby/object:Gem::Version
54
54
  version: '1.5'
55
+ - !ruby/object:Gem::Dependency
56
+ name: net-dns
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - "~>"
60
+ - !ruby/object:Gem::Version
61
+ version: '0.8'
62
+ type: :runtime
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - "~>"
67
+ - !ruby/object:Gem::Version
68
+ version: '0.8'
55
69
  - !ruby/object:Gem::Dependency
56
70
  name: rest-client
57
71
  requirement: !ruby/object:Gem::Requirement
58
72
  requirements:
59
- - - ! '>='
73
+ - - ">="
60
74
  - !ruby/object:Gem::Version
61
75
  version: '0'
62
76
  type: :runtime
63
77
  prerelease: false
64
78
  version_requirements: !ruby/object:Gem::Requirement
65
79
  requirements:
66
- - - ! '>='
80
+ - - ">="
67
81
  - !ruby/object:Gem::Version
68
82
  version: '0'
69
83
  - !ruby/object:Gem::Dependency
70
84
  name: json
71
85
  requirement: !ruby/object:Gem::Requirement
72
86
  requirements:
73
- - - ! '>='
87
+ - - ">="
74
88
  - !ruby/object:Gem::Version
75
89
  version: '0'
76
90
  type: :runtime
77
91
  prerelease: false
78
92
  version_requirements: !ruby/object:Gem::Requirement
79
93
  requirements:
80
- - - ! '>='
94
+ - - ">="
81
95
  - !ruby/object:Gem::Version
82
96
  version: '0'
83
97
  description: Enricher, the IP and URL data enhancer
@@ -87,7 +101,7 @@ executables: []
87
101
  extensions: []
88
102
  extra_rdoc_files: []
89
103
  files:
90
- - .gitignore
104
+ - ".gitignore"
91
105
  - Gemfile
92
106
  - LICENSE
93
107
  - README.md
@@ -96,9 +110,13 @@ files:
96
110
  - lib/enricher.rb
97
111
  - lib/enricher/bgpranking.rb
98
112
  - lib/enricher/bogon.rb
113
+ - lib/enricher/cdn.rb
99
114
  - lib/enricher/encoder.rb
100
115
  - lib/enricher/exceptions.rb
116
+ - lib/enricher/ipvoid.rb
117
+ - lib/enricher/resolver.rb
101
118
  - lib/enricher/version.rb
119
+ - lib/enricher/virustotal.rb
102
120
  - lib/vash.rb
103
121
  - log/enricher.log
104
122
  homepage: https://github.com/shadowbq/enricher
@@ -111,12 +129,12 @@ require_paths:
111
129
  - lib
112
130
  required_ruby_version: !ruby/object:Gem::Requirement
113
131
  requirements:
114
- - - ! '>='
132
+ - - ">="
115
133
  - !ruby/object:Gem::Version
116
134
  version: '0'
117
135
  required_rubygems_version: !ruby/object:Gem::Requirement
118
136
  requirements:
119
- - - ! '>='
137
+ - - ">="
120
138
  - !ruby/object:Gem::Version
121
139
  version: '0'
122
140
  requirements: []