iZsh-wwmd 0.2.19

data/History.txt

@@ -0,0 +1,24 @@
+== 0.2.17 / 2009-06-22
+
+* lots happening between here and .9
+* viewstate refactor complete
+* clean up page/page.rb
+* cleaning up page/headers.rb
+* cleaning up page/scrape.rb
+* FormArray refactor includes the form action (full URL)
+  * page.submit(page.get_form)
+* still bugs in URLParse but hunting them down throw by throw
+* remove broken NTLM (preserve auth header warnings)
+* remove WWMDConfig in favor of WWMD module methods but preserve old class for back compat
+* add some burp helpers
+  * Page#from_paste (take entire request into Page and turn off cookies)
+  * burp log parsing coming
+* Curb includes http_put (with header munging bug so careful)
+* internal monkey patch for Curb to do arbitrary verb tampering (not here yet)
+* add String#pbcopy
+* move lots of things around for clarity during refactor
+* refactor progressing but still unstable (2.0.16 gem including viewstate is good to go)
+
+== 0.2.9 / 2009-05-05
+
+* bonesify

data/README.rdoc

@@ -0,0 +1,87 @@
+== PARDON OUR DUST
+
+WWMD is currently in the throes of major cleanup and refactoring.
+
+0.2.17 should be stable.
+
+The viewstate tools can be had by themselves by using:
+
+  require 'wwmd/viewstate'
+
+We appreciate your patience.
+
+  <;'"}()[]>{  XSSFish says, "Swim wif me"
+
+== DESCRIPTION:
+
+WWMD was originally intended to provide a console helper tool for 
+conducting web application security assessments (which is something I 
+find myself doing alot of).  I've spent alot of time and had alot of 
+success writing application specific fuzzers + scrapers to test with.  
+WWMD provides a base of useful code to help you work with web sites both 
+in IRB and by writing scripts that can be as generic or as application 
+specific as you choose.
+
+There's alot of helpful stuff crammed in here and its usage has evolved 
+alot.  It's not intended to replace, remove or be better than any of the 
+tools you currently use.  In fact, WWMD works best *with* the tools you 
+currently use to get stuff done.  You get convenience methods for 
+getting, scraping, spidering, decoding, decrypting and munging user 
+inputs, pages and web applications.
+
+It doesn't try to be smart.  That's up to you.
+
+What's here is the basic framework for getting started.  There's a raft 
+of cookbook scripts and examples that are coming soon so make sure you 
+check the wiki regularly.
+
+== REQUIREMENTS:
+
+* rubygems
+* ruby-debug
+* curb (taf2-curb located here on github)
+* nokogiri >= 1.3.2
+* hpricot (not used by default)
+* htmlentities
+
+== INSTALL
+
+==== gem installation
+
+WWMD is available as a gem from github:
+
+    gem sources -a http://gems.github.com #(you only have to do this once)
+    gem install mtracy-wwmd
+
+=== manual installation
+
+fetch the repository from github and add path/to/wwmd/lib to your RUBYLIB 
+environment variable
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2008,2009 Michael Tracy <mtracy@matasano.com>
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+## Blah blah blah
+

data/Rakefile

@@ -0,0 +1,34 @@
+# Look in the tasks/setup.rb file for the various options that can be
+# configured in this Rakefile. The .rake files in the tasks directory
+# are where the options are used.
+
+begin
+  require 'bones'
+  Bones.setup
+rescue LoadError
+  begin
+    load 'tasks/setup.rb'
+  rescue LoadError
+    raise RuntimeError, '### please install the "bones" gem ###'
+  end
+end
+
+ensure_in_path 'lib'
+require 'wwmd'
+
+task :default => 'spec:run'
+
+PROJ.name = 'wwmd'
+PROJ.authors = 'Michael L. Tracy'
+PROJ.email = 'mtracy@matasano.com'
+PROJ.url = 'http://github.com/miketracy/wwmd/tree/master'
+PROJ.version = WWMD::VERSION
+#PROJ.rubyforge.name = 'wwmd'
+
+PROJ.spec.opts << '--color'
+
+depend_on 'ruby-debug'
+depend_on 'curb'
+depend_on 'nokogiri'
+
+# EOF

data/examples/config_example.yaml

@@ -0,0 +1,24 @@
+#---
+:base_url: "http://www.example.com"
+:header_file: "./HEADERS.default" # argv
+:username: "username" # argv
+:password: "password" # argv
+
+# opts for spider (only spider local urls)
+:spider_local_only: true
+
+# opts for curl object
+#	set max_redirects and follow_location (follows 302s)
+:follow_location: true
+:max_redirects: 20
+
+# --use_proxy=host:port overrides both of these settings
+# to use a proxy (I use burp and so should you)
+:use_proxy: false
+:proxy_url: "localhost:8080"
+
+# cookies (where are we going to save our cookies?)
+:enable_cookies: true
+:cookiejar: "./__cookiejar"
+
+#+++

data/examples/wwmd_example.rb

@@ -0,0 +1,73 @@
+#!/usr/bin/env ruby
+require 'rubygems'
+require 'wwmd'
+include WWMD
+
+$stop = lambda { Debugger.breakpoint; Debugger.catchpoint }
+
+module WWMD
+	class Page
+		# here we add directly to Page.login instead of creating an outside
+		# helper class.  Normally we create a mixin script for this.
+		def login
+			self.get(self.opts[:base_url])             ;# GET the main page redirects to /login
+			form = self.get_form                       ;# get the login form
+			if form.nil? then                          ;# did we actually get a form?
+				puts "WARN: No login form on base page"
+				return (self.logged_in = false)
+			end
+			form.set("name",self.opts[:username])      ;# set login form variables from config
+			form.set("password",self.opts[:password])
+			self.url = self.action                     ;# set the url to submit to to the form action
+			self.submit(form)                          ;# submit the form
+
+			# perform some check to make sure we aren't still on the login page
+			# (this naively checks to make sure we don't have any password fields on the current page
+			self.logged_in = (self.search("//input[@type='password']").size == 0)
+		end
+	end
+end
+
+# parse options and load configuration file
+inopts = WWMDConfig.parse_opts(ARGV)
+conf = ARGV[0] || "./config_example.yaml"
+opts = WWMDConfig.load_config(conf)
+inopts.each_pair { |k,v| opts[k] = v }
+$opts = opts
+
+# create our Page object and name it page
+page = Page.new(opts)
+page.scrape.warn = false  ;# don't complain about not overwriting scrape
+
+# move our spider object up here
+spider = page.spider
+
+# output current configuration
+puts "current configuration:\n"
+page.opts.each_pair { |k,v|
+	if k == :password then
+		puts "#{k} :: ********"
+	else
+		puts "#{k} :: #{v}"
+	end
+}
+puts "\n"
+
+# use the Helper method to login to the application
+if page.opts[:use_auth] then
+	page.login
+	if page.logged_in? then
+		puts "logged in as #{opts[:username]}"
+	else
+		puts "WARN: could not log in" if !page.logged_in?
+	end
+else
+	page.get opts[:base_url]
+end
+
+# report our current location and let's drop to irb with
+# our whole context complete
+puts "current location: #{page.current}"
+puts "enter \"irb\" to go to the console"
+
+$stop.call

data/lib/wwmd.rb

@@ -0,0 +1,78 @@
+# third-party
+require 'rubygems'
+require 'ruby-debug'
+require 'curb'
+require 'yaml'
+require 'fileutils'
+require 'base64'
+require 'optparse'
+require 'digest'
+require 'uri'
+require 'htmlentities'
+require 'nkf'
+require 'rexml/document'
+
+module WWMD
+
+  # :stopdoc:
+  VERSION = "0.2.19"
+  PARSER = :nokogiri  # :nokogiri || :hpricot
+  LIBPATH = ::File.expand_path(::File.dirname(__FILE__)) + ::File::SEPARATOR
+  PATH = ::File.dirname(LIBPATH) + ::File::SEPARATOR
+  # :startdoc:
+
+  # Returns the version string for the library.
+  #
+  def self.version
+    VERSION
+  end
+
+  # Returns the library path for the module. If any arguments are given,
+  # they will be joined to the end of the libray path using
+  # <tt>File.join</tt>.
+  #
+  def self.libpath( *args )
+    args.empty? ? LIBPATH : ::File.join(LIBPATH, args.flatten)
+  end
+
+  # Returns the lpath for the module. If any arguments are given,
+  # they will be joined to the end of the path using
+  # <tt>File.join</tt>.
+  #
+  def self.path( *args )
+    args.empty? ? PATH : ::File.join(PATH, args.flatten)
+  end
+
+  # Utility method used to require all files ending in .rb that lie in the
+  # directory below this file that has the same name as the filename passed
+  # in. Optionally, a specific _directory_ name can be passed in such that
+  # the _filename_ does not have to be equivalent to the directory.
+  #
+  def self.require_all_libs_relative_to( fname, dir = nil )
+    dir ||= ::File.basename(fname, '.*')
+    search_me = ::File.expand_path(
+        ::File.join(::File.dirname(fname), dir, '**', '*.rb'))
+
+    Dir.glob(search_me).sort.each do |rb|
+      next if rb =~ /html2text_/
+      require rb
+    end
+  end
+
+end  # module WWMD
+
+WWMD.require_all_libs_relative_to(__FILE__)
+
+# special case parser
+
+if WWMD::PARSER == :nokogiri
+  require 'nokogiri'
+  WWMD::HDOC = Nokogiri::HTML
+  require 'wwmd/page/html2text_nokogiri'
+else
+  require 'hpricot'
+  WWMD::HDOC = Hpricot
+  require 'wwmd/page/html2text_hpricot'
+end
+
+# EOF

data/lib/wwmd/class_extensions.rb

@@ -0,0 +1,2 @@
+Dir.glob(::File.join(::File.dirname(__FILE__),"class_extensions/","*.rb")).each { |rb| require rb }
+

data/lib/wwmd/class_extensions/extensions_base.rb

@@ -0,0 +1,235 @@
+require 'htmlentities'
+
+=begin rdoc
+let's re-open everything!
+=end
+
+require 'uri'
+
+class Numeric
+  # return binary representation of <tt>length</tt> size padded with \x00
+  #  length:  length in bytes to return (padded with least signficant \x00
+  #  reverse: reverse the byte order
+  def to_bin (len,rev = false)
+    str = ""
+    bignum = self
+    1.upto(len) do |i|
+      str << (bignum & 0xFF).to_n8
+      bignum = bignum >> 8  
+    end
+    return str.reverse if rev
+    return str
+  end
+
+  # integer to ip address
+  def int_to_ip
+    [24, 16, 8, 0].map { |b| (self >> b) & 255 }.join('.')
+  end
+
+  # integer to mac address [uses ':' as delimiter]
+  def int_to_mac
+    [40,32,24,16,8,0].map { |b| ((self >> b) & 255).to_s(16).rjust(2,"0") }.join(":")
+  end
+end
+
+class String
+
+  def strip_up
+    self.gsub(/[^\x20-\x7e,\n]/,"").gsub(/^\n/,"")
+  end
+
+  # ip address to int
+  def ip_to_int
+    self.split('.').inject(0) { |a,e| (a << 8) + e.to_i }
+  end
+
+  # mac address to int [uses ':' as delimiter]
+  def mac_to_int
+    self.split(':').inject(0) { |a,e| (a << 8) + e.to_i(16) }
+  end
+
+  # return true or false for <tt>string.match</tt>  
+  def contains?(rexp)
+    return !self.match(rexp).nil?
+  end
+
+  # strip the string and return true if empty
+  def empty?
+    return self.strip == ''
+  end
+
+  # return everything in the string (url) before the first get param
+  ## "http://foo.bar.com/page.asp?somearg=foo&otherarg=bar".clip  
+  ## => "http://foo.bar.com/page.asp"
+  def clip(pref="?")
+    if (v = self.index(pref))
+      return self[0..(v-1)]
+    end
+    return self
+  end
+
+  # return everything in the string (url) after the first get parameter
+  # without the leading '?'
+  #
+  # pass true as the second param to also get back the ?
+  ## "http://foo.bar.com/page.asp?somearg=foo&otherarg=bar".clop 
+  ## => "somearg=foo&otherarg=bar"
+  def clop(pref="?",preftoo=false)
+    (preftoo == false) ? add = "" : add = pref
+    if (v = self.index(pref))
+      return add + self[(v+1)..-1]
+    end
+    return nil
+  end
+
+  def clopp; self.clop("?",true); end #:nodoc:
+
+  def clopa
+    return [self.clip,self.clop]
+  end
+
+  alias_method :clipa, :clopa
+
+  # File.dirname with a trailing slash
+  def dirname
+    return self if self.match(/\/$/)
+    File.dirname(self) + "/"
+  end
+
+  # File.basename
+  def basename(ext=nil)
+    if ext
+      File.basename(self,ext)
+    else
+      File.basename(self)
+    end
+  end
+
+  def extname
+    self.split('.').last
+  end
+
+  # write string to passed filename
+  # if filename is nil? will raise an error
+  def write(fname=nil)
+    raise "filename required" if fname.nil?
+    File.write(fname,self)
+    return fname
+  end
+
+  # parse passed GET param string into a form and return the FormArray object
+  def to_form
+    if self.split("\n").size > 1
+      return self.to_form_from_show
+    end
+    ret = FormArray.new
+    self.split("&").each do |x|
+      y = x.split("=",2)
+      ret.extend!(y[0].to_s,y[1].to_s)
+    end
+    return ret
+  end
+
+  def to_form_from_show
+    self.split("\n").map { |a|
+      key,val = a.split("=",2)
+      key = key.split(" ")[-1]
+      val = val.strip if val
+      ["#{key}=#{val}"]
+    }.join("&").to_form.squeeze_keys!
+  end
+
+  def mform
+    return self.gsub("\n","").to_form
+  end
+
+  def to_form_from_req
+#    self.split("\x0d\x0a\x0d\x0a")[1].to_form
+    self.split("\n\n")[1].to_form
+  end
+  alias_method :to_ffr, :to_form_from_req
+
+  # create filename from url changing "/" to "_"
+  def to_fn(ext=nil)
+    ret = self.clip.split("/")[3..-1].join("_")
+    ret += ".#{ext}" if not ext.nil?
+    return ret
+  end
+
+  # strip html tags from string
+  def strip_html
+    self.gsub(/<\/?[^>]*>/, "")
+  end
+
+  # range or int
+  def head(c=5)
+    if c.kind_of?(Range) then
+      range = c
+    else
+      range = (0..(c - 1))
+    end
+    self.split("\n")[range].join("\n")
+  end
+
+  # return a literal regexp object for this string
+  #
+  # escape regexp operators
+  def to_regexp
+    return Regexp.new(self.gsub(/([\[\]\{\}\(\)\*\$\?])/) { |x| '\\' + x })
+  end
+
+  # check if this string is a guid
+  def is_guid?
+    begin
+      Guid.from_s(self)
+    rescue => e
+      return false
+    end
+    return true
+  end
+
+  def md5
+    Digest::MD5.digest(self).hexify
+  end
+
+  def sha1
+    Digest::SHA1.digest(self).hexify
+  end
+
+  def sha256
+    Digest::SHA256.digest(self).hexify
+  end
+
+  def sha512
+    Digest::SHA512.digest(self).hexify
+  end
+
+  def pbcopy
+    IO.popen('pbcopy', 'r+') { |c| c.print self }
+  end
+end
+
+class Array
+  # grep each element of an array for the passed regular expression
+  # and return an Array of matches
+  # (only works one deep)
+  def each_grep(regex)
+    ret = []
+    self.each { |e| ret << e.grep(regex) }
+    return ret
+  end
+
+  # join the array with "\n" and write to a file
+  def to_file(filename)
+    File.write(filename,self.join("\n"))
+  end
+end
+
+class File
+  # write string to file
+  def self.write(filename,contents)
+    fout = File.open(filename,"w")
+    fout.print contents
+    fout.close
+  end
+end