wmap 2.5.5 → 2.5.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9b7418c10fec65599e729f381e47814776b8e9641a3e1d27d5b11084e2d87f3d
-  data.tar.gz: 8534aa535bbba34f58fab3edd1e73cad1b85d2c95d294d2e6bbd894960c1f885
+  metadata.gz: 6afbdfcdf343c39f15a1af467762bffe90bc12f05b7a7eb1341ef2cacf7e561f
+  data.tar.gz: 95e039be31be2277fcc8e225106c365b1b7b1694d4db372a6f211d552c3d1162
 SHA512:
-  metadata.gz: 8992d8b673bf2fddfb5a1097b26a947fb109fb68bccf677310fee4b7e4530358ef48da75eb128f70c0f8e609914c91f3c5f0348de9417d38683db1ed1de1fd3e
-  data.tar.gz: c8664f0216f971f48f454bdced1a5566cb0feaa699c2a366fd34e72f84b6744e54e623d3c959e1369d2aea751eb5cbbd92e2d3bf78dc621821d5f5b953e2dd7f
+  metadata.gz: b522ee4f4830ed50fb4dac0760cf6b15d84d8b791bd6c1593cdc51042747d002b14f46b1173a40fef7c952fb1256c019b118ec47b6c691c3b4fd27f5f0d3d4b3
+  data.tar.gz: 5f0bd019eac486a744dd96c9d36928d27ba2607cfe1d9e6834189d756f85e08b4deec09b45b89bfc7e779e385f44ec862f6e516d04e98b4f3e31b09c458ea76e

data/{README.rdoc → README.md}

@@ -1,32 +1,38 @@
-= OWASP WebMapper Ruby API README
+[<img src='/wmap_logo.jpg' width='350' height='350'>](https://github.com/yangsec888/wmap)
+=====================
 
+- [What's this program for?](#whats-this-program-for)
+- [WMAP in Motion](#wmap-in-motion)
+- [Installation](#installation)
+- [Before Using This Program](#before-using-this-program)
+- [More Document(s)](#more-documents)
+- [Program Version](#program-version)
+- [Author Contact](#author-contact)
+- [Bug Report or Feature Request](#bug-report-or-feature-request)
+- [Legal Disclaimer](#legal-disclaimer)
 
-== What's this program for?
+# OWASP Web Mapper README
+
+
+## What's this program for?
 This program is designed for the web application asset discovery and tracking. It was originally developed to
 to cover the gaps of a similar commercial product. Over the time it grows to be a more capable and complete replacement (IMHO).
 
 
-== WMAP in Motion
+## WMAP in Motion
 Use the demo web app build on top of wmap gem: http://wmap.io
 
 
-== Program Version
-The latest release is version 2.4.5. as of fall 2018. Please refer to the CHANGELOG.md for the program's history information.
-
-
-== Author Contact
-This program is designed and developed by Yang Li. You can reach him by Email: <yang.li@owasp.org>
-
-
-== Installation
+## Installation
 To take full power of this program, you would need an *nix flavor machine with direct Internet access. I have installed it successfully on both Mac and Linux machines. You'll also need the Ruby environment being setup properly. The easiest way to install OWASP Web Mapper is by using Ruby Gems. You can install it from command line:
-
+```sh
   gem install wmap
+```
 
-== Specific Installation Problem with Nokogiri
+### Specific Installation Problem with Nokogiri
 Nokogiri is a native xml/html parser used by the project. It's fast and powerful. However, it comes with pitfall of installation problem around building native extension for your environment. Please refer to this page for trouble-shooting tip (http://www.nokogiri.org/tutorials/installing_nokogiri.html).
 
-== Dependency
+### Dependency
 You need the Ruby 2.1.0 or above in order to use this program. In my test environment, I was able to set it up with <a href="https://rvm.io/">RVM</a>. Please refer to this page for more installation information: https://www.ruby-lang.org/en/documentation/installation/
 
 In addition, the following Ruby GEM dependency are needed by different components of this software. The should be installed automatically:
@@ -48,31 +54,32 @@ In case you want to install the above gems separately, use the command below:
 
       gem install dnsruby geoip minitest net-ping netaddr nokogiri css_parser open_uri_redirections openssl parallel whois httpclient
 
-== Ruby-whois Gem Patches
+### Ruby-whois Gem Patches
 This software depends on a patched version of Ruby gem ruby-whois (http://www.ruby-whois.org/) for the domain whois lookup feature. For better result, you could manually add the patches into your local whois gem installation directory as shown below:
-
+```sh
       cp whois_patches/* [Your_ruby_whois_gem_path]/whois/lib/whois/record/parser/
+```
 
 Or you can directly download the branched whois gem from this repository - https://github.com/yangsec888/whois
 
 
-== Before Using This Program
+### Before Using This Program
 You need to define a scope for the program to run successful. The scope includes both your legitimate Internet domain, and your public
 network block in the CIDR format.
 
 To add your Internet domain into the scope, use the build-in shell command below:
-
+```sh
      trust XYZ.COM
-
+```
 To add your public network block into the scope (note current support of IPv4 only):
-
+```sh
      trust x.x.x.x/x
+```
 
-
-== Automatic Discovery and Tracking
-
+### Automatic Discovery and Tracking
+```sh
     wmap <seed file | target host | target url | target IP or network cidr>
-
+```
 The above utility is intelligent enough to take argument as either a seed file, or a string such as a host, an IP, a network block, or a URL. The new discoveries will be automatically tracked in the data file 'lib/wmap/data/target_sites'.
   Note: seed file - mix of url, cidr and domain seed, one entry per line.
 				url seed - known URL(s) for further discovery via the web crawler.
@@ -80,26 +87,33 @@ The above utility is intelligent enough to take argument as either a seed file,
 				domain seed - validated internet domain to be used for DNS record brute-forcing; it is also used to validate the ownership of found web service.
 
 
-== Dump Out Discovery Database
+### Dump Out Discovery Database
 You can dump out the program output by using the build-in utility 'wdump' as shown below:
-
+```sh
     wdump [output file name from you]
-
+```
 The above utility will dump out the discovery database into a single file as program output. Currently, the supported file format is Comma-separated Value (.csv) and Extensible Markup Language (.xml)
 
 
-== More Usage Cases:
+### More Usage Cases:
 There are more examples under the 'demos' folder of this package. The examples show how to use the 'wmap' API to get your job done easily. Please check out the code - they should be easy and straightforward to be understood.
 
 
-== More Document(s):
+## More Document(s):
 The software comes with the Ruby doc during your installation as shown above. For your convenience, the Ruby doc is also distributed with this software. You can navigate to the 'doc' folder of your local installation, and click the 'index.html' to open the start page in your favorite browser. You can also download the wmap-x.x.x.rdoc.zip documentation package alone from GitHub, unzip and open the doc/index.html in your browser.
 
 If you need additional documentation / information other than this README file and the Ruby document package, please be patient - as I'm still working on it :)
 
-== How do I report the bugs, or maybe require some new features?
-Contact the author Sam Li directly at email 'yang.li@owasp.org'.
+
+## Program Version
+The latest release is version [2.5.5+](version.txt). as of fall 2018. Please refer to the [CHANGELOG.md](CHANGELOG.md) for more history information.
+
+
+## Author Contact
+This program is designed and developed by Yang Li. You can reach him by Email: <yang.li@owasp.org>
+## Bug Report or Feature Request?
+Contact the author Sam (Yang) Li directly at email 'yang.li@owasp.org'.
 
 
-== Legal Disclaimer:
+## Legal Disclaimer:
 This software is provided strictly 'as-if' without any implied warranty. You're free to copy or modify the codes anyway you want - a reference back to this software will be appreciated. Please refer to the 'LICENSE.txt' file for more information.

data/bin/wmap

@@ -5,22 +5,25 @@
 # Usage: wmap <Target Host | URL | IP | CIDR | or a seed file with any of the above combo> <Optional Discovery Result Directory>
 require "wmap"
 
+# program helper
 def print_usage
 	abort "Program to perform website asset discovery and tracking. \nUsage: wmap <Target Host | URL | IP | CIDR | or a seed file with any of the above combo> <Optional Discovery Result Directory>"
 end
-
-# preparing - spit out the program banner
+# print program banner
 puts Wmap.banner
+
+# Preparing - check out the working logs directory
 if ARGV.length == 2
 	# Log to the instance running directory
-	Log_dir=File.dirname(__FILE__)+'/../logs/'+ARGV[1]
+	Log_dir = Pathname.new(ARGV[1]).join('logs')
 else
 	# Log the command entry
-	Log_dir=File.dirname(__FILE__)+'/../logs/'
+	Log_dir=Pathname.new(Gem.loaded_specs['wmap'].full_gem_path).join('logs')
 end
 Dir.mkdir(Log_dir) unless Dir.exist?(Log_dir)
 
-Wmap.wlog("Execute the command: wmap #{ARGV[0]}","wmap",Log_dir+"wmap.log")
+# Start wmap logging
+Wmap.wlog("Execute the command: wmap #{ARGV[0]}","wmap",Log_dir.join("wmap.log").to_s)
 print_usage unless (ARGV.length==1 or ARGV.length==2)
 urls = Array.new
 # first step - construct the host list
@@ -64,38 +67,16 @@ else
 	print_usage
 end
 
-# second step - update the hosts repository
-if ARGV.length == 1
-	puts puts "Invoke the HostTracker."
-	host_tracker = Wmap::HostTracker.instance
-	host_tracker.verbose=true
-elsif ARGV.length == 2
-	puts "Invoke the HostTracker with optional directory setter."
-	host_tracker = Wmap::HostTracker.instance
-	host_tracker.verbose=true
-	host_tracker.data_dir = ARGV[1]
-else
-	aborts "Error firing up HostTracker instance!"
-end
-hosts.uniq!
-if hosts.size > 0
-	hostnames=hosts.dup.delete_if { |h| host_tracker.is_ip?(h) }
-	if hostnames.size > 0
-		puts "Update the local hosts data repository."
-		new_hosts=host_tracker.adds(hostnames)
-		host_tracker.save! if new_hosts.size>0
-	end
-end
-host_tracker=nil
 
-# third step - port discovery on the above host list, and to build the URL seeds
+# second step - port discovery on the above host list, and to build the URL seeds
 puts "Build up URL list for the web crawler ..."
 urls0=scanner.scans(hosts)
 urls+=urls0
 urls.uniq!
 scanner=nil
 
-# fourth step - crawling on the URL seeds
+
+# third step - crawling on the URL seeds
 if ARGV.length == 1
 	puts "Fire up the crawler."
 	crawler = Wmap::UrlCrawler.new(:verbose=>false)
@@ -125,14 +106,16 @@ else
 	dis_sites.keys.map {|x| puts x}
 end
 
-# fifth step - trace the discovery results into a local log file for debugging and other purposes
+
+# fourth step - trace the discovery results into a local log file for debugging and other purposes
 Wmap.wlog(dis_urls.keys, "wmap", Log_dir+"discovered_urls.log") unless dis_urls.empty?
 Wmap.wlog(dis_sites.keys, "wmap", Log_dir+"discovered_sites.log") unless dis_sites.empty?
 #crawler.wlog(c_start.keys,Log_dir+"crawler.log")
 #crawler.wlog(c_done.keys,Log_dir+"crawler.log")
 crawler=nil
 
-# sixth step - save discovery results into the inventory data repository
+
+# fifth step - save discovery results into the inventory data repository
 case dis_sites.keys
 when nil,[]
 	puts "No new site found. There is no change to the site tracking data repository. "
@@ -145,6 +128,8 @@ else
 		puts "Start the SiteTracker with the optional directory setter. "
 		inventory=Wmap::SiteTracker.instance
 		inventory.data_dir = ARGV[1]
+		inventory.sites_file = inventory.data_dir + "sites"
+		inventory.load_site_stores_from_file
 	else
 		aborts "Error firing up SiteTracker instance!"
 	end
@@ -153,3 +138,32 @@ else
 	inventory=nil
 	puts "Done! New found sites are successfully saved. " if new_sites.size > 0
 end
+
+
+# seventh step - update the hosts repository
+if ARGV.length == 1
+	puts puts "Invoke the HostTracker."
+	host_tracker = Wmap::HostTracker.instance
+	host_tracker.verbose=true
+elsif ARGV.length == 2
+	puts "Invoke the HostTracker with optional directory setter."
+	host_tracker = Wmap::HostTracker.instance
+	host_tracker.verbose=true
+	host_tracker.data_dir = ARGV[1]
+	host_tracker.hosts_file = host_tracker.data_dir + "hosts"
+	host_tracker.load_known_hosts_from_file
+else
+	aborts "Error firing up HostTracker instance!"
+end
+new_hosts = dis_sites.keys.map {|x| host_tracker.url_2_host(x)}
+hosts += new_hosts
+hosts.uniq!
+if hosts.size > 0
+	hostnames=hosts.dup.delete_if { |h| host_tracker.is_ip?(h) }
+	if hostnames.size > 0
+		puts "Update the local hosts data repository with: #{hostnames}"
+		new_hosts=host_tracker.adds(hostnames)
+		host_tracker.save! if new_hosts.size>0
+	end
+end
+host_tracker=nil

data/lib/wmap/domain_tracker/sub_domain.rb

@@ -17,8 +17,7 @@ class SubDomain < Wmap::DomainTracker
 	include Wmap::Utils
 	include Singleton
 
-	attr_accessor :verbose, :domains_file, :max_parallel, :data_dir
-	attr_reader :known_internet_sub_domains
+	attr_accessor :verbose, :sub_domains_file, :max_parallel, :data_dir, :known_internet_sub_domains
 
 	# Set default instance variables
 	def initialize (params = {})
@@ -26,111 +25,102 @@ class SubDomain < Wmap::DomainTracker
 		@data_dir=params.fetch(:data_dir, File.dirname(__FILE__)+'/../../../data/')
 		@max_parallel=params.fetch(:max_parallel, 40)
 		# Hash table to hold the trusted domains
-		@file_sub_domains=@data_dir + 'sub_domains'
-		File.write(@file_sub_domains, "") unless File.exist?(@file_sub_domains)
-		@known_internet_sub_domains=load_domains_from_file(@file_sub_domains) #unless @known_internet_sub_domains.size>0
-		end
+		@sub_domains_file=params.fetch(:sub_domains_file, @data_dir + 'sub_domains')
+		File.write(@sub_domains_file, "") unless File.exist?(@sub_domains_file)
+		@known_internet_sub_domains=load_domains_from_file(@sub_domains_file) #unless @known_internet_sub_domains.size>0
+	end
 
 	# 'setter' to add sub-domain entry to the cache one at a time
 	def add(sub)
 		puts "Add entry to the local sub domain cache table: #{sub}" if @verbose
-		begin
-			record=Hash.new
-			sub=sub.strip.downcase
-			if @known_internet_sub_domains.key?(sub)
-				puts "Skip on known sub-domain: #{sub}" if @verbose
-				return nil
-			end
-			if zone_transferable?(sub)
-				record[sub]=true
-			else
-				record[sub]=false
-			end
-			puts "Adding new record into the data store: #{record}" if @verbose
-			@known_internet_sub_domains.merge!(record)
-			return record
-		rescue => ee
-			puts "Exception on method #{__method__} for #{sub}: #{ee}" if @verbose
+		record=Hash.new
+		sub=sub.strip.downcase
+		if @known_internet_sub_domains.key?(sub)
+			puts "Skip on known sub-domain: #{sub}" if @verbose
+			return nil
 		end
+		if zone_transferable?(sub)
+			record[sub]=true
+		else
+			record[sub]=false
+		end
+		puts "Adding new record into the data store: #{record}" if @verbose
+		@known_internet_sub_domains.merge!(record)
+		return record
+	rescue => ee
+		puts "Exception on method #{__method__} for #{sub}: #{ee}" if @verbose
 	end
 
 	# 'setter' to add domain entry to the cache in batch (from a list)
 	def bulk_add(list, num=@max_parallel)
 		puts "Add entries to the local domains cache table from list: #{list}" if @verbose
-		begin
-			results=Hash.new
-			domains=list
-			if domains.size > 0
-				Parallel.map(list, :in_processes => num) { |target|
-					add(target)
-				}.each do |process|
-					if process.nil?
-						next
-					elsif process.empty?
-						#do nothing
-					else
-						results.merge!(process)
-					end
+		results=Hash.new
+		domains=list
+		if domains.size > 0
+			Parallel.map(list, :in_processes => num) { |target|
+				add(target)
+			}.each do |process|
+				if process.nil?
+					next
+				elsif process.empty?
+					#do nothing
+				else
+					results.merge!(process)
 				end
-				@known_internet_sub_domains.merge!(results)
-				puts "Done loading entries."
-				return results
-			else
-				puts "Error: no entry is loaded. Please check your list and try again."
 			end
+			@known_internet_sub_domains.merge!(results)
+			puts "Done loading entries."
 			return results
-		rescue => ee
-			puts "Exception on method #{__method__}: #{ee}" if @verbose
+		else
+			puts "Error: no entry is loaded. Please check your list and try again."
 		end
+		return results
+	rescue => ee
+		puts "Exception on method #{__method__}: #{ee}" if @verbose
 	end
 	alias_method :adds, :bulk_add
 
 	# Procedures to identify sub-domain from the hosts store
 	def update_from_host_store!
 		puts "Invoke internal procedures to update the sub-domain list from the host store."
-		begin
-			# Step 1 - obtain the latest sub-domains
-			my_tracker = Wmap::HostTracker.instance
-			my_tracker.data_dir=@data_dir
-			subs = my_tracker.dump_sub_domains - [nil,""]
-			my_tracker = nil 
-			# Step 2 - update the sub-domain list
-			unless subs.empty?
-				#subs.map { |x| self.add(x) unless domain_known?(x) }
-				self.bulk_add(subs,@max_parallel)
-			end
-			puts "Update discovered sub-domains into the store: #{@known_internet_sub_domains}"
-			self.save!(file_domains=@file_sub_domains, domains=@known_internet_sub_domains)
-		rescue Exception => ee
-			puts "Exception on method #{__method__}: #{ee}" if @verbose
-			return nil
+		# Step 1 - obtain the latest sub-domains
+		my_tracker = Wmap::HostTracker.instance
+		my_tracker.data_dir=@data_dir
+		subs = my_tracker.dump_sub_domains - [nil,""]
+		my_tracker = nil
+		# Step 2 - update the sub-domain list
+		unless subs.empty?
+			#subs.map { |x| self.add(x) unless domain_known?(x) }
+			self.bulk_add(subs,@max_parallel)
 		end
+		puts "Update discovered sub-domains into the store: #{@known_internet_sub_domains}"
+		self.save!(file_domains=@file_sub_domains, domains=@known_internet_sub_domains)
+	rescue Exception => ee
+		puts "Exception on method #{__method__}: #{ee}" if @verbose
+		return nil
 	end
 	alias_method :update!, :update_from_host_store!
 
 	# Save the current domain hash table into a file
 	def save_sub_domains_to_file!(file_domains=@file_sub_domains, domains=@known_internet_sub_domains)
 		puts "Saving the current domains cache table from memory to file: #{file_domains} ..." if @verbose
-		begin
-			timestamp=Time.now
-			f=File.open(file_domains, 'w')
-			f.write "# Local domains file created by class #{self.class} method #{__method__} at: #{timestamp}\n"
-			f.write "# domain name, free zone transfer detected?\n"
-			domains.keys.sort.map do |key|
-				if domains[key]
-					f.write "#{key}, yes\n"
-				else
-					f.write "#{key}, no\n"
-				end
+		timestamp=Time.now
+		f=File.open(file_domains, 'w')
+		f.write "# Local domains file created by class #{self.class} method #{__method__} at: #{timestamp}\n"
+		f.write "# domain name, free zone transfer detected?\n"
+		domains.keys.sort.map do |key|
+			if domains[key]
+				f.write "#{key}, yes\n"
+			else
+				f.write "#{key}, no\n"
 			end
-			f.close
-			puts "Domain cache table is successfully saved: #{file_domains}"
-		rescue => ee
-			puts "Exception on method #{__method__}: #{ee}" if @verbose
 		end
+		f.close
+		puts "Domain cache table is successfully saved: #{file_domains}"
+	rescue => ee
+		puts "Exception on method #{__method__}: #{ee}" if @verbose
 	end
 	alias_method :save!, :save_sub_domains_to_file!
-end
 
 	# Print summary report on all known / trust domains in the domain cache table
 	def print_known_sub_domains
@@ -143,3 +133,4 @@ end
 
 end
 end
+end