wmap 2.4.4 → 2.4.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 5694ee5d1b8c7c612253fe814a4c829ea30f0107
-  data.tar.gz: f20a472f8b02efb83c918d2a8359d09adeed6bd9
+SHA256:
+  metadata.gz: 781e42205ee21710dc610778284fb6ab3fc4644598632d204db6bdc291af216f
+  data.tar.gz: c2c1c527be70b2452b1aa66ea2138af78a094ae82eb8ddfde15a2ec09df30700
 SHA512:
-  metadata.gz: 1598c1da69120421bd68ba0dc92730e07e7a099206255ada23a55ac4db6e79410af0b9a4184fb56779cef9ab5f5cfe887ca22d7e0505117d08450c36293a59b7
-  data.tar.gz: 23532584bc9deec9fb6fc76a1537c474b7ee05e0e74d27d9c79d2d4ef880d10a5a77eb13c8d6e7f04b58b32339e20bb3d989002a4319fee36ad29a4b938cf74b
+  metadata.gz: ab3231e48a2d65f777afa36a9efb28bc741dfe79f0a10aae64d18ad6861206ca209984635b57ffe863bfa4a602d07cc79ad472561c3eccf1fb527973ef15bd0b
+  data.tar.gz: 55ead86d6498b73e0b2511518dcd3cdc69468456fc3525ff523b03a38b31bbadd18185e94c8afea777c3dd3f0d943b7ee1868482bbd2badecb4b339ad14eb6f5

data/README.rdoc

@@ -6,6 +6,10 @@ This program is designed for the web application asset discovery and tracking. I
 to cover the gaps of a similar commercial product. Over the time it grows to be a more capable and complete replacement (IMHO).
 
 
+== Wmap in Motion
+Use the demo web app build on top of wmap gem: http://wmap.io/
+
+
 == Program Version
 The latest release is Beta version 1.5.x as of fall 2014. Please refer to the CHANGELOG.md for the program's history information.
 
@@ -19,36 +23,36 @@ To take full power of this program, you would need an *nix flavor machine with d
 
   gem install wmap-x.x.x.gem --no-rdoc
 
+== Specific Installation Problem with Nokogiri
+Nokogiri is a native xml/html parser used by the project. It's fast and powerful. However, it comes with pitfall of installation problem around building native extension for your environment. Please refer to this page for trouble-shooting tip (http://www.nokogiri.org/tutorials/installing_nokogiri.html).
 
 == Dependency
 You need the Ruby 1.9.2 or above in order to use this program. In my test environment, I was able to set it up with RVM. Please refer to this page for more installation information: http://www.ruby-lang.org/en/downloads/
 
- In addition, the following Ruby GEM dependency are needed by different components of this software:
-      require "digest/md5"
+In addition, the following Ruby GEM dependency are needed by different components of this software. The should be installed automatically:
       require "dnsruby"
       require "geoip"
       require "minitest/autorun"
-      require "net/http"
       require "net/ping"
       require "netaddr"
       require "nokogiri"
-      require "open_uri_redirections"
+      require "css_parser"
       require "openssl"
-      require "open-uri"
+      require "open_uri_redirections"
       require "parallel"
-      require "resolv"
-      require "singleton"
-      require "uri"
       require "whois"
       require 'httpclient'
-      require 'nokogiri'
-      require 'open-uri'
- To install "uri" gem for example, use the command below:
-      $ gem install uri
+
+
+In case you want to install the above gems separately, use the command below:
+
+      gem install dnsruby geoip minitest net-ping netaddr nokogiri css_parser open_uri_redirections openssl parallel whois httpclient
 
 == Ruby-whois Gem Patches
-This software depends on a patched version of Ruby gem ruby-whois-2.7.0 (http://www.ruby-whois.org/) for the domain whois lookup feature. For better result, you could manually add the patches into your local whois gem installation directory as shown below:
-  $ cp whois_patches/* [Your_ruby_whois_gem_path]/whois/lib/whois/record/parser/
+This software depends on a patched version of Ruby gem ruby-whois (http://www.ruby-whois.org/) for the domain whois lookup feature. For better result, you could manually add the patches into your local whois gem installation directory as shown below:
+
+      cp whois_patches/* [Your_ruby_whois_gem_path]/whois/lib/whois/record/parser/
+
 Or you can directly download the branched whois gem from this repository - https://github.com/yangsec888/whois
 
 
@@ -57,15 +61,17 @@ You need to define a scope for the program to run successful. The scope includes
 network block in the CIDR format.
 
 To add your Internet domain into the scope, use the build-in shell command below:
-  $ trust XYZ.COM
 
-To add your public network block into the scope:
-  $ trust x.x.x.x/x
+     trust XYZ.COM
+
+To add your public network block into the scope (note current support of IPv4 only):
+
+     trust x.x.x.x/x
 
 
 == Automatic Discovery and Tracking
 
-  $ wmap <seed file | target host | target url | target IP or network cidr>
+    wmap <seed file | target host | target url | target IP or network cidr>
 
 The above utility is intelligent enough to take argument as either a seed file, or a string such as a host, an IP, a network block, or a URL. The new discoveries will be automatically tracked in the data file 'lib/wmap/data/target_sites'.
   Note: seed file - mix of url, cidr and domain seed, one entry per line.
@@ -76,7 +82,8 @@ The above utility is intelligent enough to take argument as either a seed file,
 
 == Dump Out Discovery Database
 You can dump out the program output by using the build-in utility 'wdump' as shown below:
-  $ wdump [output file name from you]
+
+    wdump [output file name from you]
 
 The above utility will dump out the discovery database into a single file as program output. Currently, the supported file format is Comma-separated Value (.csv) and Extensible Markup Language (.xml)
 
@@ -91,7 +98,7 @@ The software comes with the Ruby doc during your installation as shown above. Fo
 If you need additional documentation / information other than this README file and the Ruby document package, please be patient - as I'm still working on it :)
 
 == How do I report the bugs, or maybe require some new features?
-Contact the author Yang Li directly at email 'yang.li@owasp.org'.
+Contact the author Sam Li directly at email 'yang.li@owasp.org'.
 
 
 == Legal Disclaimer:

data/bin/RHPG

@@ -0,0 +1,85 @@
+#!/usr/bin/env ruby
+# Executable to lookup then merge site tech details into the RHPG asset spreadsheet in CSV format only
+#
+## Usage: RHPG [RHPG.csv]
+require "wmap"
+require "csv"
+include Wmap::Utils
+
+def print_usage
+	puts "Program to lookup then merge the site details into RHPG asset spreadsheet. \nUsage: RHPG [RHPG.csv]"
+end
+
+def site_lookup(domain)
+  tracker=Wmap::SiteTracker.new(:verbose=>false)
+  #first order search
+  tracker.known_sites.each do |key,val|
+    if key.include?(domain.strip.downcase) && key.include?("https")
+      tracker=nil
+      return [key] + val.values
+    end
+  end
+  #second order search
+  tracker.known_sites.each do |key,val|
+    if key.include?(domain.strip.downcase)
+      tracker=nil
+      return [key] + val.values
+    end
+  end
+  tracker=nil
+  return [nil]*9
+end
+
+def wp_site_lookup(domain)
+  tracker=Wmap::WpTracker.new(:verbose=>false)
+	# first order
+	tracker.known_wp_sites.each do |key,val|
+		if key.include?(domain.strip.downcase) && val
+			ver=tracker.wp_ver(key)
+			tracker=nil
+			return [val,ver]
+		end
+	end
+  # second order
+  tracker.known_wp_sites.each do |key,val|
+    if key.include?(domain.strip.downcase) && key.include?("https")
+      tracker=nil
+      return [val,nil]
+    end
+  end
+  # third order
+  tracker.known_wp_sites.each do |key,val|
+    if key.include?(domain.strip.downcase)
+      tracker=nil
+      return [val,nil]
+    end
+  end
+  tracker=nil
+  return [nil,nil]
+end
+
+puts Wmap.banner
+print_usage
+
+# open output file to write
+CSV.open("output.csv", "wb") do |csv|
+  cnt=1
+  # open RHPG input file to read
+  CSV.foreach(ARGV[0]) do |row|
+    puts "Processing row #{cnt}"
+    #puts row.inspect
+    my_row=Array.new
+    if cnt > 1
+      if is_domain?(row[0])
+        my_row = row + site_lookup(row[0]) + wp_site_lookup(row[0])
+      else
+        my_row = row + [nil]*10
+      end
+    else
+      my_row = row + ["Website","Primary IP","Port","Hosting Status","Server","Response Code","MD5 Finger-print","Redirection","Timestamp", "WordPress", "WordPress Version"]
+    end
+    cnt+=1
+    csv << my_row
+  end
+  puts "All done. "
+end

data/bin/trust

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 require "wmap"
-# Executable to add seed entry into ring of the trust. I.E. the trusted domain or CIDR 
+# Executable to add seed entry into ring of the trust. I.E. the trusted domain or CIDR
 
 def print_usage
 	puts "Program to add trust authority entry. Usage: trust [domain|CIDR]"
@@ -11,15 +11,15 @@ print_usage
 Log_dir=File.dirname(__FILE__)+'/../logs/'
 Wmap.wlog("Execute the command: trust #{ARGV[0]}","trust",Log_dir+"wmap.log")
 
-dt=Wmap::DomainTracker.instance
-ct=Wmap::CidrTracker.new
+dt=Wmap::DomainTracker.new
+ct=Wmap::CidrTracker.new(:verbose=>true)
 abort "Incorrect program argument! Proper usage: trust [domain | netblock]" unless ARGV.length==1 && (dt.is_fqdn?(ARGV[0]) || ct.is_cidr?(ARGV[0]))
 
 puts "Start the baptizing process ..."
 
 # Add entry into the local repository
 
-if dt.is_domain?(ARGV[0]) 
+if dt.is_domain?(ARGV[0])
 	result=dt.add(ARGV[0])
 	unless result.nil?
 		dt.save!
@@ -27,7 +27,7 @@ if dt.is_domain?(ARGV[0])
 	end
 end
 dt=nil
-	
+
 if ct.is_cidr?(ARGV[0])
 	result=ct.add(ARGV[0])
 	unless result.nil?

data/bin/trusts

@@ -0,0 +1,38 @@
+#!/usr/bin/env ruby
+# Executable to add seed entries into ring of the trust. I.E. the trusted domain or CIDR
+require "wmap"
+include Wmap::Utils
+def print_usage
+	puts "Program to add trust authority entries. Usage: trust [domain|CIDR list in a file]"
+end
+
+puts Wmap.banner
+print_usage
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: trust #{ARGV[0]}","trust",Log_dir+"wmap.log")
+
+dt=Wmap::DomainTracker.new
+ct=Wmap::CidrTracker.new(:verbose=>true)
+abort "Incorrect program argument! Proper usage: trust [domain | netblock]" unless ARGV.length==1 && (File.exist?(ARGV[0]))
+
+puts "Start the baptizing process ..."
+
+file_2_list(ARGV[0]).map do |target|
+	# Add entry into the local repository
+	if dt.is_domain?(target)
+		result=dt.add(target)
+		unless result.nil?
+			dt.save!
+			puts "Domain #{target} is successfully baptized!"
+		end
+	elsif ct.is_cidr?(target)
+		result=ct.add(target)
+		unless result.nil?
+			ct.save!
+			puts "Net block #{target} is successfully baptized!"
+		end
+	end
+end
+
+dt=nil
+ct=nil

data/bin/updateAll

@@ -19,7 +19,7 @@ else
 	abort "You got it. Mission is successfully aborted. "
 end
 # Update sub-domain table
-sd=Wmap::DomainTracker::SubDomain.instance
+sd=Wmap::DomainTracker::SubDomain.new
 sd.update_from_host_store!
 subs=sd.known_internet_sub_domains.keys
 sd=nil
@@ -29,29 +29,25 @@ bruter=Wmap::DnsBruter.new
 sub_hosts=bruter.brutes(subs).values.flatten
 
 # Update primary host store
-ph=Wmap::HostTracker::PrimaryHost.instance
+ph=Wmap::HostTracker::PrimaryHost.new
 ph.update_from_site_store!
 ph.refresh_all
 ph.save!
 ph=nil
 
 # Update host store
-h=Wmap::HostTracker.instance
+h=Wmap::HostTracker.new
 h.refresh_all
 h.adds(sub_hosts)
 h.save!
 h=nil
 
 # Update site store
-st=Wmap::SiteTracker.instance
+st=Wmap::SiteTracker.new
 st.refresh_all
-dt=Wmap::SiteTracker::DeactivatedSite.instance
+dt=Wmap::SiteTracker::DeactivatedSite.new
 ds=dt.known_sites.keys
 st.adds(ds)					#double-check the de-activated sites in case the site is back on-line again
 st.save!
 st=nil
 dt=nil
-
-
-
-

data/bin/wadds

@@ -11,7 +11,7 @@ print_usage
 Log_dir=File.dirname(__FILE__)+'/../logs/'
 Wmap.wlog("Execute the command: wadds #{ARGV[0]}","wadds",Log_dir+"wmap.log")
 
-st=Wmap::SiteTracker.instance
+st=Wmap::SiteTracker.new
 abort "Incorrect program argument!" unless ARGV.length==1 && File.exist?(ARGV[0])
 
 # Evaluate the argument and update the data store accordingly

data/bin/wmaps

@@ -0,0 +1,24 @@
+#!/usr/bin/env ruby
+# script to automate the new site discovery through by crawling all unique sites in the site store
+require "wmap"
+require "parallel"
+
+def wmap_worker(domain)
+   cmd = "wmap " + domain
+   puts "wmap discovery on domain: ", domain
+   system(cmd)
+end
+
+
+tracker=Wmap::DomainTracker.new
+=begin
+Parallel.map(tracker.known_internet_domains.keys, :in_processes => 10) { |target|
+  puts "Working on #{target} ..." if @verbose
+  wmap_worker(target)
+}
+=end
+tracker.known_internet_domains.keys.map do |domain|
+  wmap_worker(domain)
+end
+
+tracker=nil