wmap 2.4.4

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 5694ee5d1b8c7c612253fe814a4c829ea30f0107
+  data.tar.gz: f20a472f8b02efb83c918d2a8359d09adeed6bd9
+SHA512:
+  metadata.gz: 1598c1da69120421bd68ba0dc92730e07e7a099206255ada23a55ac4db6e79410af0b9a4184fb56779cef9ab5f5cfe887ca22d7e0505117d08450c36293a59b7
+  data.tar.gz: 23532584bc9deec9fb6fc76a1537c474b7ee05e0e74d27d9c79d2d4ef880d10a5a77eb13c8d6e7f04b58b32339e20bb3d989002a4319fee36ad29a4b938cf74b

data/CHANGELOG.md

@@ -0,0 +1,141 @@
+##
+# Wmap
+#
+- A pure Ruby library for Internet web application discovery and tracking.
+-
+- Copyright (c) 2012-2015 Yang Li <yang.li@owasp.org>
+
+
+# Change-log
+
+
+## Mile-stones
+
+- July 2015: Move the project under OWASP
+- November 2014: Re-name from web_discovery to wmap, re-factor the code base to better scale up
+	across the board
+- May 2013: First release of web_discovery gem package 1.1.0
+- December 2012: Alpha Release 0.1.0, with everything in one file for fast prototype purpose
+
+
+## Backward Incompatibilities
+
+- List of features that are backward incompatible:
+
+
+## Beta Release 1.x
+
+- 07/25/2015  Clean up for OWASP release.
+-	04/14/2015	Fix bug in 'trust'/'distrust' utilities, so that they could properly handle CIDR as input format.
+-	04/02/2015	Removing tracking of the 'un-reponsive' http site under 'Wmap::SiteTracker' class.
+-	03/30/2015	Fix a bug in 'bulk_refresh' method under 'Wmap::SiteTracker' class.
+-	03/24/2015	Implement 'wcheck' executable support to improve user experience.
+-	03/23/2015	Implement 'wadds' executable support to improve user experience.  
+-	02/19/2015	Bug fix on url_2_site method to handle rare cases containing "-" or "?" special chars.
+-	02/18/2015	Implement the tcp port-scanner executable 'wscan'.
+-	02/17/2015	A bug fix on the Wmap::Utils.normalize_url method, where the rare case of trailing dot after hostname could
+-				  properly removed as well.
+-	02/11/2015	A bug fix on the Wmap::SiteTracker.get_prim_uniq_sites method, where host resolved to multiple IPs could
+-				  cause exception in the rare case.
+-	02/10/2015	Implement the 'singleton' module in the Wmap::HostTracker::PrimaryHost class
+-	02/05/2015	Implement singleton pattern on the Wmap::DomainTracker::SubDomain, Wmap::SiteTracker::DeactivatedSite class.  
+-	02/03/2015	Implement singleton pattern on the logger module, re-organize the log file structure and location.
+-	01/30/2015	Implement the singleton pattern on the Wmap::SiteTracker, Wmap::DomainTracker class.
+-	01/27/2015	Separate the logger sub-module and implement the singleton pattern on the logger.
+-	01/24/2015	Implement the 'singleton' module in the Wmap::HostTracker class, in order to avoid race condition
+-				  under the parallel engine
+-	01/08/2015	Implement the Ruby MiniTest Unit-test frame-work, with the first unit test file 'utils_test.rb'
+-	12/15/2014	Replace the instance variables @known_hosts, @known_sites with class variables
+-				 across all modules,i.e. enforcing the singleton pattern to avoid race condition under parallel execution.
+-	11/17/2014	Re-factor the code Wmap::SiteTracker, optimize the 'dump' and 'dump_xml' methods.
+-	11/14/2014	Change the package name from "WebDiscovery" version 1.5.3 to "Wmap" to simple reason, start with version 1.0  
+-	11/13/2014	Add XML support as the program output format, i.e. 'save_uniq_sites_xml' method for WebDiscovery::SiteTracker class
+-	11/01/2014	Add 'add' and 'delete' methods for WebDiscovery::CidrTracker class, in order to make it self-contained and user friendly.
+-	10/31/2014	Add executables (trust, distrust etc..) under bin directory, so that the application is more user friendly
+-	10/28/2014	Add parallel support in the HostTracker, SubDomain classes.
+-	10/27/2014	Add 'prime', 'wdump' bin executables, to manually set the prime host, and to dump out unique sites
+-				 in the site tracking data repository respectively.
+-	10/21/2014	Re-implement bulk_refresh method under WebDiscovery::SiteTracker class,
+-				 with parallel engine support.
+-	09/29/2014	Add 'save' method to WebDiscovery::UrlCrawler class.
+-	09/08/2014	Support parallel and add 'checks' method to WebDiscovery::UrlChecker class, in order to
+-				 scale up.
+-	08/04/2014	Add brute_all method for WebDiscovery::DnsBruter class.
+-	07/21/2014	Add consistency checks for WebDiscovery::SiteTracker::Dump method.
+-	07/08/2014	Add stop_hostname method in the class WebDiscovery::HostTracker; enhance the DnsBruter methods
+-				 accordingly.
+-	05/19/2014	Add site_ip_known? method in the class WebDiscovery::SiteTracker
+-	03/31/2014	Add deduplicate procedures in the class WebDiscovery::HostTracker::PrimaryHost
+-	02/10/2014	Add additional class WebDiscovery::DomainTracker::SubDomain to better manage sub-domains
+-				 make changes in DomainTracker, HostTracker, and executable 'wd' accordingly
+-	02/07/2014	Add additional class WebDiscovery::SiteTracker::DeactivatedSite and logic to record
+-				 the decommissioned or no longer accessible site
+-	01/21/2014	Introduce 'hostname_mutation' method into WebDiscovery::DnsBruter class
+-	01/15/2014	Modify WebDiscovery::Utils::UrlMagic class and add URL 'normalize' method
+-	01/14/2014	Modify WebDiscovery::UrlCrawler class and set hard stop limit of crawler time-out to
+-				1 0 minutes per instance.
+-   01/06/2014	Revamp WebDiscovery::UrlCrawler class, to make it more readable and scalable.
+-   12/19/2013	Add new method 'SiteTracker.site_check', to pull out a record from the site store.
+-   12/04/2013	Extend the usage of the 'parallel' to the 'SiteTracker.refresh_all' method, in order to
+-				 scale up the completion time.
+-   11/06/2013	Include GeoIPLite into the gem pack and modify the links under GeoIPTracker class.
+-   10/29/2013	Fix a small bug in the 'HostTracker.bulk_add' method
+-   10/21/2013	Change the data structure of domain_tracker class, to include open zone transfer
+-                information.
+-   10/09/2013	Add additional logic to handle Dnsruby::ServFail type error intelligently in order to
+-                optimize the DnsBruter speed.
+-   10/08/2013	Add additional logic to avoid repeating crawling the same link by different child processes.
+-   10/04/2013	Add additional logic to profile the web server to improving the crawler speed
+-   09/30/2013	Add additional logic to eliminate crawling duplicate sites for the multi-threaded crawlers.
+-   09/23/2013  Optimize the DNS bruter code, as we have a large number of internet domains/sub-domains
+-				 that cause overflow when using the array concatenation method.
+-   09/09/2013  Fix of error handling of "Connection reset by peer" in 'url_checker' class,
+-				 along with other minors  
+-   07/16/2013  Add an intelligent network profiler to maximize the port scanner performance.  
+-   06/24/2013  Add support to map CN in the ssl sites into the primary host table.
+-   06/11/2013  Fix some minor bugs within the dns_bruter class
+-   05/29/2013  Implement Google search scraper for Google search engine discovery.
+-   05/20/2013  Support sub-domain identification and brute-forcing by: a) add 'get_subdomains' method in
+-				 the 'DomainRoot' module; b) add 'dump_sub_domains' method in the 'HostTracker' class;
+-				 c) implement 'dns_brute_subdomains' method in the 'DnsBruter' class
+-   05/16/2013  Add 'server' type into 'SiteTracker' data structure, implement a simple 'search' method.
+-   05/15/2013  Implement the 'get_server_header' method for the UrlChecker class
+-   05/08/2013  Implement the 'resolve_ip_sites' method for the SiteTracker class
+-   05/07/2013  Implement GeoIP tracker class WebDiscovery::GeoIPTracker, which wrap around the  
+-				 Ruby GeoIP 1.1.0 gem - http://geoip.rubyforge.org/
+-   05/06/2013  Add methods in the WebDiscovery::Whois class to extract netname and description
+-				 when performing whois lookup for an IP address
+-   05/02/2013  Change data structure for the 'SiteTracker' class, add service port field
+-				 for better tracking purpose; add an ASCII art banner
+-   04/25/2013  Add method to retrieve common name from a server cert in the 'UrlChecker' class
+-   04/23/2013  Fully test out the port scanner, DNS bruter and the main executable;
+-				 change README.txt to README.rdoc; create Rakefile
+-   04/18/2013  patch main executable 'webDiscovery' to make it flexible for the users
+-   04/15/2013  Finish major re-haul of the code base. Now it's broke into OO code-let
+-	03/11/2013	Add support for country code second level domain lookup
+-	02/25/2013	Implement new methods to handle the internet domain seed file
+-	12/12/2012	Re-factor the crawler code; add new feature of the hosts cache table for
+-				 local domain name (reverse) lookup  
+-	11/30/2012	Phase out the dependency on the 'dig' command; replace it with the native
+-				 Ruby 'resolv', and 'dnsruby' modules instead
+-	11/27/2012	Re-factor 'get_domain_root' method, fix a bug there
+-	11/21/2012	Implement the 'whois' wrapper for the domain research
+-	11/16/2012	Implement @know_cidr_blks class hash for discovered app labelling
+-	11/15/2012	Optimize the algorithm used by the method 'ip_known?'
+-	11/14/2012	Fix small bug in method 'host_2_ips'
+-	11/13/2012	Fix small bugs handling url case-insensitivity
+-	11/08/2012	Add methods for host discovery via open zone transfer
+-	11/07/2012	Add the dns_brute_force method
+-	11/06/2012	Re-factor the crawler code
+-	11/05/2012	Bug fix on the class attributes access
+-	10/28/2012	Bug fix on the reverse-dns lookup method
+-	10/26/2012	Implement a web crawler for website crawling and link extraction
+-	10/24/2012	Re-factor the port discovery method for better performance
+-	10/23/2012	Implement the port discovery method
+-	10/22/2012	Implement a simple HTTP service detection method
+-	10/21/2012	Implement a simple TCP port scanner, and SSL socket detection methods
+-	10/20/2012	Implement the IP address validation process
+-	10/19/2012	Reimplement the URL status check method
+-	10/18/2012	Implement proprietary URI manipulation methods
+-	10/17/2012	Refine and reimplement the DNS query methods and process
+-	10/16/2012	Start from the drawing board, prototype BESTwebDiscovery class.

data/LICENSE.txt

@@ -0,0 +1,15 @@
+Copyright (c) 2012-2015 Yang Li <yang.li8@owasp.org>
+
+Apache License 2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/README.rdoc

@@ -0,0 +1,98 @@
+= OWASP WebMapper Ruby API README
+
+
+== What's this program for?
+This program is designed for the web application asset discovery and tracking. It was originally developed to
+to cover the gaps of a similar commercial product. Over the time it grows to be a more capable and complete replacement (IMHO).
+
+
+== Program Version
+The latest release is Beta version 1.5.x as of fall 2014. Please refer to the CHANGELOG.md for the program's history information.
+
+
+== Author Contact
+This program is designed and developed by Yang Li. You can reach him by Email: <yang.li@owasp.org>
+
+
+== Installation
+To take full power of this program, you would need an *nix flavor machine with direct Internet access. I have installed it successfully on both Mac and Linux machines. You'll also need the Ruby environment being setup properly. The easiest way to install OWASP Web Mapper is by using Ruby Gems. Download the latest gem 'wmap-x.x.x.gem' into the local file system. Then install it from command line there:
+
+  gem install wmap-x.x.x.gem --no-rdoc
+
+
+== Dependency
+You need the Ruby 1.9.2 or above in order to use this program. In my test environment, I was able to set it up with RVM. Please refer to this page for more installation information: http://www.ruby-lang.org/en/downloads/
+
+ In addition, the following Ruby GEM dependency are needed by different components of this software:
+      require "digest/md5"
+      require "dnsruby"
+      require "geoip"
+      require "minitest/autorun"
+      require "net/http"
+      require "net/ping"
+      require "netaddr"
+      require "nokogiri"
+      require "open_uri_redirections"
+      require "openssl"
+      require "open-uri"
+      require "parallel"
+      require "resolv"
+      require "singleton"
+      require "uri"
+      require "whois"
+      require 'httpclient'
+      require 'nokogiri'
+      require 'open-uri'
+ To install "uri" gem for example, use the command below:
+      $ gem install uri
+
+== Ruby-whois Gem Patches
+This software depends on a patched version of Ruby gem ruby-whois-2.7.0 (http://www.ruby-whois.org/) for the domain whois lookup feature. For better result, you could manually add the patches into your local whois gem installation directory as shown below:
+  $ cp whois_patches/* [Your_ruby_whois_gem_path]/whois/lib/whois/record/parser/
+Or you can directly download the branched whois gem from this repository - https://github.com/yangsec888/whois
+
+
+== Before Using This Program
+You need to define a scope for the program to run successful. The scope includes both your legitimate Internet domain, and your public
+network block in the CIDR format.
+
+To add your Internet domain into the scope, use the build-in shell command below:
+  $ trust XYZ.COM
+
+To add your public network block into the scope:
+  $ trust x.x.x.x/x
+
+
+== Automatic Discovery and Tracking
+
+  $ wmap <seed file | target host | target url | target IP or network cidr>
+
+The above utility is intelligent enough to take argument as either a seed file, or a string such as a host, an IP, a network block, or a URL. The new discoveries will be automatically tracked in the data file 'lib/wmap/data/target_sites'.
+  Note: seed file - mix of url, cidr and domain seed, one entry per line.
+				url seed - known URL(s) for further discovery via the web crawler.
+				cidr seed - list of known network blocks, for discovering web service via port scanning; it is also used to validate if the web service has a known IP (internal hosted).
+				domain seed - validated internet domain to be used for DNS record brute-forcing; it is also used to validate the ownership of found web service.
+
+
+== Dump Out Discovery Database
+You can dump out the program output by using the build-in utility 'wdump' as shown below:
+  $ wdump [output file name from you]
+
+The above utility will dump out the discovery database into a single file as program output. Currently, the supported file format is Comma-separated Value (.csv) and Extensible Markup Language (.xml)
+
+
+== More Usage Cases:
+There are more examples under the 'demos' folder of this package. The examples show how to use the 'wmap' API to get your job done easily. Please check out the code - they should be easy and straightforward to be understood.
+
+
+== More Document(s):
+The software comes with the Ruby doc during your installation as shown above. For your convenience, the Ruby doc is also distributed with this software. You can navigate to the 'doc' folder of your local installation, and click the 'index.html' to open the start page in your favorite browser. You can also download the wmap-x.x.x.rdoc.zip documentation package alone from GitHub, unzip and open the doc/index.html in your browser.
+
+If you need additional documentation / information other than this README file and the Ruby document package, please be patient - as I'm still working on it :)
+
+== How do I report the bugs, or maybe require some new features?
+Contact the author Yang Li directly at email 'yang.li@owasp.org'.
+
+
+== Legal Disclaimer:
+This software is provided strictly 'as-if' without any implied warranty. You're free to copy or modify the codes anyway you want - a reference back to this software will be appreciated. Please refer to the 'LICENSE.txt' file for more information.

data/TODO

@@ -0,0 +1,13 @@
+# The following improvements would be nice to have:
+
+
+1. Add unit test cases for all major modules / classes (Note: Jan. 2015, Wmap::Utils module unit tests added).
+2. Add Google search capability (Note: Oct. 2013, partial done, need Google API licensing /fee in order for the full implementation test. )
+3. Better documentations / man pages for executables / operation manuals, in addition to the 'README' and 'rdoc'
+4. Port the local data repository to a relational DB (for support of multi user sessions in the future)
+5. Implement a RoR web interface for better user experience; with standard SOAP API as a web service. 
+6. Implement a report engine for better metric reporting.
+7. Improve the crawler, such as supporting link extraction in AJAX, FLASH and other rich HTML5 format
+8. Improve the build-in port scanner 'wscan'. So it could eventually replace 'nmap' in the larget scale network blocks scanning process.
+9. Add vulnerability scanner engine.
+10.Add vulnerability life cycle management.

data/bin/deprime

@@ -0,0 +1,21 @@
+#!/usr/bin/env ruby
+# Executable to remove a fully qualified domain name into the prime host tracking data repository
+# This is useful as a self-correction mechanism to flag out unique website in a constant way
+require "wmap"
+
+def print_usage
+	puts "Program to remove a primary host entry in the local data repository. Usage: deprime [fully qualified domain name]"
+end
+
+puts Wmap.banner
+print_usage
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: deprime #{ARGV[0]}","deprime",Log_dir+"wmap.log")
+
+# Update primary host store
+ph=Wmap::HostTracker::PrimaryHost.instance
+abort "Incorrect program argument! Proper Usage: deprime [fully qualified domain name]" unless ARGV.length==1 && ph.is_fqdn?(ARGV[0])
+
+ph.delete(ARGV[0])
+ph.save!
+ph=nil

data/bin/distrust

@@ -0,0 +1,38 @@
+#!/usr/bin/env ruby
+# Executable to remove entry from the ring of the trust. I.E. the trusted domain or CIDR 
+require "wmap"
+
+def print_usage
+	puts "Program to remove trust authority entry. Usage: distrust [domain | netblock]"
+end
+
+puts Wmap.banner
+print_usage
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: distrust #{ARGV[0]}","distrust",Log_dir+"wmap.log")
+
+dt=Wmap::DomainTracker.instance
+ct=Wmap::CidrTracker.new(:verbose=>false)
+abort "Incorrect program argument! Proper Usage: distrust [domain | netblock]" unless ARGV.length==1 && (dt.is_fqdn?(ARGV[0]) || ct.is_cidr?(ARGV[0]))
+
+puts "Start the demonizing ..."
+
+
+if dt.is_domain?(ARGV[0]) 
+	result=dt.delete(ARGV[0])
+	unless result.nil?
+		dt.save!
+		abort "Domain #{ARGV[0]} is successfully demonized!"
+	end
+end
+dt=nil
+
+if ct.is_cidr?(ARGV[0].to_s)
+	result=ct.delete(ARGV[0].to_s)
+	unless result.nil?
+		ct.save!
+		abort "Net block #{ARGV[0]} is successfully demonized!"
+	end
+end
+ct=nil
+

data/bin/googleBot

@@ -0,0 +1,23 @@
+#!/usr/bin/env ruby
+# Using the Google scraper module to query Google engines on different geographic locations, then work through the google results to update site store
+require 'wmap'
+
+def print_usage
+	puts "Program to discovery website assets from Google search engines. Usage: googleBot"
+end
+
+puts Wmap.banner
+print_usage
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: googleBot","googleBot",Log_dir+"wmap.log")
+
+abort "Incorrect program argument - no argument needed! Proper Usage: googleBot " unless ARGV.length==0
+
+puts "Start the Google search scraper. This may take a while, please be patient ..."
+dis=Wmap::GoogleSearchScraper.new
+dis.workers
+sites=dis.discovered_sites_from_scraper.keys
+puts "Update Google search results into the data repository:"
+k=Wmap::SiteTracker.instance
+k.adds(sites)
+k.save!

data/bin/prime

@@ -0,0 +1,21 @@
+#!/usr/bin/env ruby
+# Executable to add a fully qualified domain name into the prime host tracking data repository
+# This is useful as a self-correction mechanism to flag out unique website in a constant way
+require "wmap"
+
+def print_usage
+	puts "Program to add a primary host entry in the local data repository. Usage: prime [hostname]"
+end
+
+puts Wmap.banner
+print_usage
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: prime #{ARGV[0]}","prime",Log_dir+"wmap.log")
+
+# Update primary host store
+ph=Wmap::HostTracker::PrimaryHost.instance
+abort "Incorrect program argument! Proper Usage: prime [fully qualified domain name]" unless ARGV.length==1 && ph.is_fqdn?(ARGV[0])
+
+ph.add(ARGV[0])
+ph.save!
+ph=nil

data/bin/refresh

@@ -0,0 +1,26 @@
+#!/usr/bin/env ruby
+# Executable to refresh a site within the tracking data repository
+# This is useful as site information may change over the time
+## Usage: refresh [a site|all]
+require "wmap"
+
+def print_usage
+	puts "Program to refresh the website entry in the local data repository. Usage: refresh [a site|all]"
+end
+
+puts Wmap.banner
+print_usage
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: refresh #{ARGV[0]}","refresh",Log_dir+"wmap.log")
+
+abort "Incorrect program argument! Proper Usage: refresh [site]" unless ARGV.length==1
+
+#puts "Captured command argument: #{ARGV[0]}"
+if ARGV[0]=="all"
+	# magic token 'all' to refresh all sites in the store
+	Wmap.refresh_all
+else
+	abort "Error: incorrect site syntax! It must be in http(s):://xxx.xxx/ format." unless Wmap::SiteTracker.instance.is_site?(ARGV[0])
+st=nil
+	Wmap.refresh(ARGV[0])
+end

data/bin/run_tests

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+# Ruby MiniTest Unit-test Auto Runner
+require "wmap"
+
+def print_usage
+	puts "Program to run all the build-in unit tests under /test. Usage: run_tests"
+end
+
+puts Wmap.banner
+print_usage
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: run_tests","run_tests",Log_dir+"wmap.log")
+
+spec = Gem::Specification.find_by_name("wmap")
+Dir[spec.gem_dir+'/test/*.rb'].map { |test| puts "Executing test file: #{test}";system('ruby', test) }
+

data/bin/spiderBot

@@ -0,0 +1,26 @@
+#!/usr/bin/env ruby
+# script to automate the new site discovery through by crawling all unique sites in the site store
+require "wmap"
+
+def print_usage
+	puts "Program to crawl all unique sites within the site store, then update the store accordingly. \nUsage: spiderBot"
+end
+
+puts Wmap.banner
+print_usage
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: spiderBot","spiderBot",Log_dir+"wmap.log")
+
+abort "Incorrect program argument - no argument need! Proper Usage: spiderBot" unless ARGV.length==0
+dis=Wmap::SiteTracker.instance
+sites=dis.get_uniq_sites
+
+crawler=Wmap::UrlCrawler.new(:max_parallel=>40)
+crawler.crawls(sites)
+v_sites=crawler.discovered_urls_by_crawler.keys.map {|x| crawler.url_2_site(x) }
+v_sites.uniq!
+
+dis.adds(v_sites)
+dis.save!
+
+

data/bin/trust

@@ -0,0 +1,38 @@
+#!/usr/bin/env ruby
+require "wmap"
+# Executable to add seed entry into ring of the trust. I.E. the trusted domain or CIDR 
+
+def print_usage
+	puts "Program to add trust authority entry. Usage: trust [domain|CIDR]"
+end
+
+puts Wmap.banner
+print_usage
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: trust #{ARGV[0]}","trust",Log_dir+"wmap.log")
+
+dt=Wmap::DomainTracker.instance
+ct=Wmap::CidrTracker.new
+abort "Incorrect program argument! Proper usage: trust [domain | netblock]" unless ARGV.length==1 && (dt.is_fqdn?(ARGV[0]) || ct.is_cidr?(ARGV[0]))
+
+puts "Start the baptizing process ..."
+
+# Add entry into the local repository
+
+if dt.is_domain?(ARGV[0]) 
+	result=dt.add(ARGV[0])
+	unless result.nil?
+		dt.save!
+		abort "Domain #{ARGV[0]} is successfully baptized!"
+	end
+end
+dt=nil
+	
+if ct.is_cidr?(ARGV[0])
+	result=ct.add(ARGV[0])
+	unless result.nil?
+		ct.save!
+		abort "Net block #{ARGV[0]} is successfully baptized!"
+	end
+end
+ct=nil

data/bin/updateAll

@@ -0,0 +1,57 @@
+#!/usr/bin/env ruby
+# the up to date program to refresh all local cache tables in one shot. Note it requires an uninterrupted Internet connection to perform the job. It also takes a long time so patient is needed. Lastly, don't forget to backup the existing 'data' folder before execute this command, cause any unexpected interruption may wreak havoc on the existing data file!!!
+
+require "wmap"
+
+puts Wmap.banner
+puts "Program to refresh the local data repository."
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: updateAll","updateAll",Log_dir+"wmap.log")
+
+abort "Incorrect program argument - no argument needed! Proper Usage: updateAll" unless ARGV.length==0
+
+puts "You're about to update Wmap data repository. It'll take a long time. And the Internet connection must be un-interrupted during the process. You're also expected to backup the data folder before proceeding. Are you ready? (Yes/No)"
+STDOUT.flush
+answer=gets.chomp
+if answer =~ /yes/i
+	puts "Start up to date process. Please be patient ..."
+else
+	abort "You got it. Mission is successfully aborted. "
+end
+# Update sub-domain table
+sd=Wmap::DomainTracker::SubDomain.instance
+sd.update_from_host_store!
+subs=sd.known_internet_sub_domains.keys
+sd=nil
+
+# Brute force sub-domains to detect sub-domain hosts
+bruter=Wmap::DnsBruter.new
+sub_hosts=bruter.brutes(subs).values.flatten
+
+# Update primary host store
+ph=Wmap::HostTracker::PrimaryHost.instance
+ph.update_from_site_store!
+ph.refresh_all
+ph.save!
+ph=nil
+
+# Update host store
+h=Wmap::HostTracker.instance
+h.refresh_all
+h.adds(sub_hosts)
+h.save!
+h=nil
+
+# Update site store
+st=Wmap::SiteTracker.instance
+st.refresh_all
+dt=Wmap::SiteTracker::DeactivatedSite.instance
+ds=dt.known_sites.keys
+st.adds(ds)					#double-check the de-activated sites in case the site is back on-line again
+st.save!
+st=nil
+dt=nil
+
+
+
+

data/bin/wadd

@@ -0,0 +1,25 @@
+#!/usr/bin/env ruby
+# Executable to add a new site into the tracking data repository
+# This is useful as a self-correction mechanism to flag out unique website in a constant way
+require "wmap"
+
+def print_usage
+	puts "Program to add an entry into the local data repository. Usage: wadd [site]"
+end
+
+puts Wmap.banner
+print_usage
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: wadd #{ARGV[0]}","wadd",Log_dir+"wmap.log")
+
+st=Wmap::SiteTracker.instance
+abort "Incorrect program argument!" unless ARGV.length==1
+
+# Evaluate the argument and update the data store accordingly
+if st.is_site?(ARGV[0])
+	st.add(ARGV[0])
+	st.save!
+	st=nil
+else
+	abort "Unknown argument format: #{ARGV[0]}"
+end

data/bin/wadds

@@ -0,0 +1,26 @@
+#!/usr/bin/env ruby
+# Executable to bulk add sites into the tracking data repository
+require "wmap"
+
+def print_usage
+	puts "Program to add sites from a file into local data repository. Usage: wadds [file_sites]"
+end
+
+puts Wmap.banner
+print_usage
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: wadds #{ARGV[0]}","wadds",Log_dir+"wmap.log")
+
+st=Wmap::SiteTracker.instance
+abort "Incorrect program argument!" unless ARGV.length==1 && File.exist?(ARGV[0])
+
+# Evaluate the argument and update the data store accordingly
+sites=st.file_2_list(ARGV[0]).map { |x| st.url_2_site(x) }
+if sites.length > 0
+	news=st.adds(sites)
+	puts news
+	st.save! if news.length>0
+	st=nil
+else
+	abort "No site entry found in file: #{ARGV[0]}. Please check your file format to ensure one site per line."
+end

data/bin/wcheck

@@ -0,0 +1,28 @@
+#!/usr/bin/env ruby
+# Executable to check the current status of a new site 
+# This is useful as a self-correction mechanism to flag out unique website in a constant way
+require "wmap"
+
+def print_usage
+	puts "Program to quick check the status of a remote URL. Usage: wcheck [URL]"
+end
+
+puts Wmap.banner
+
+Log_dir=File.dirname(__FILE__)+'/../logs/'
+Wmap.wlog("Execute the command: wcheck #{ARGV[0]}","wcheck",Log_dir+"wmap.log")
+
+checker=Wmap::UrlChecker.new(:verbose=>false)
+unless ARGV.length==1 
+	print_usage
+	abort "Incorrect program argument!" 
+end
+
+# Evaluate the argument and update the data store accordingly
+if checker.is_url?(ARGV[0])
+	status=checker.url_worker(ARGV[0])
+	puts status
+else
+	print_usage
+	abort "Invalid URL format: #{ARGV[0]}"
+end