arachni 0.2.2.1 → 0.2.2.2

data/CHANGELOG.md

@@ -1,6 +1,36 @@
 
 # ChangeLog
 
+## Version 0.2.2.2 _(Under development)_
+- Added "arachni_web_autostart" under bin -- Automatically starts all systems required by the WebUI and makes shutting down everything easier too (Original by: Brandon Potter <bpotter8705@gmail.com>)
+- Overrided Nokogiri to revert to UTF-8 when it comes across an unknown charset instead of throwing exceptions
+- Dependency versions are now defined explicitly [Issue #23]
+- Updated to Sinatra v1.2.1
+- HTTP
+   - Disabled peer verification on SSL [Issue #19]
+   - Replaced callbacks with the new _Observable_ mixin (also updated components to use the new conventions)
+- WebUI
+   - Plug-in options are preserved [Issue #19]
+   - Check-all now skips disabled checkboxes
+   - Report info is stored in a database [Issue #19]
+   - Reports are now displayed in descending order based on scan completion datetime [Issue #19]
+   - Any existing reports will be migrated into the new database [Issue #19]
+- XMLRPC service
+   - Fixed segfault on forced shutdown when spider-first was enabled
+- Plug-ins
+   - AutoLogin now registers its results
+- Reports -- Added formatters for the AutoLogin [Issue #19] and Profiler plug-ins
+   - HMTL
+      - Fixed exception on empty issue list
+      - Fixed encoding exceptions (cheers to Chris Weber <chris@casaba.com>)
+- Path extractors
+   - Generic -- fixed error on invalid encoding sequences
+- Modules
+   - Recon
+       - Directory listing -- Now skips non-200 pages because it used to log false positives on redirections
+- Plug-ins
+   - Added Profiler -- Performs taint analysis (with benign inputs) and response time analysis
+
 ## Version 0.2.2.1 _(February 13, 2011)_
 - Web UI v0.1-pre (Utilizing the Client - Dispatch-server XMLRPC architecture) (**New**)
    - Basically a front-end to the XMLRPC client

data/CONTRIBUTORS.md

@@ -4,6 +4,7 @@ These are the people that helped improve Arachni either by submitting code, sugg
 
 - [Matías Aereal Aeón](http://mfsec.com.ar/), **Arachni's official tester**.
 - [Christos Chiotis](mailto:chris@survivetheinternet.com) for designing the new HTML report template.
+- [Brandon Potter](mailto:bpotter8705@gmail.com) for the original "arachni_web_autostart" script
 - [Steve Pinkham](http://github.com/spinkham) for beta testing and patches.
 - [Aung Khant](mailto:aungkhant@yehg.net) for general suggestions.
 

data/README.md

@@ -1,7 +1,8 @@
 # Arachni - Web Application Security Scanner Framework
-**Version**:     0.2.2.1<br/>
-**Homepage**:     [http://github.com/zapotek/arachni](http://github.com/zapotek/arachni)<br/>
-**News**:     [http://trainofthought.segfault.gr/category/projects/arachni/](http://trainofthought.segfault.gr/category/projects/arachni/)<br/>
+**Version**:     0.2.2.2<br/>
+**Homepage**:     [http://arachni.segfault.gr](http://arachni.segfault.gr)<br/>
+**Blog**:         [http://trainofthought.segfault.gr/category/projects/arachni/](http://trainofthought.segfault.gr/category/projects/arachni/)<br/>
+**Github page**:  [http://github.com/zapotek/arachni](http://github.com/zapotek/arachni)<br/>
 **Documentation**:     [http://github.com/Zapotek/arachni/wiki](http://github.com/Zapotek/arachni/wiki)<br/>
 **Code Documentation**:     [http://zapotek.github.com/arachni/](http://zapotek.github.com/arachni/)<br/>
 **Google Group**: [http://groups.google.com/group/arachni](http://groups.google.com/group/arachni)<br/>
@@ -159,14 +160,15 @@ The analyzer can graciously handle badly written HTML code due to a combination
  - Plug-ins are framework demi-gods, they have direct access to the framework instance.
  - Can be used to add any functionality to Arachni.
  - Currently available plugins:
-    - Passive Proxy
+    - Passive Proxy -- Analyzes requests and responses between the web app and the browser assisting in AJAX audits, logging-in and/or restricting the scope of the audit
     - Form based AutoLogin
     - Dictionary attacker for HTTP Auth
     - Dictionary attacker for form based authentication
-    - Cookie collector
+    - Profiler -- Performs taint analysis (with benign inputs) and response time analysis
+    - Cookie collector -- Keeps track of cookies while establishing a timeline of changes
     - Healthmap -- Generates sitemap showing the health of each crawled/audited URL
     - Content-types -- Logs content-types of server responses aiding in the identification of interesting (possibly leaked) files
-    - WAF (Web Application Firewall) Detector
+    - WAF (Web Application Firewall) Detector -- Establishes a baseline of normal behavior and uses rDiff analysis to determine if malicious inputs cause any behavioral changes
     - MetaModules -- Loads and runs high-level meta-analysis modules pre/mid/post-scan
        - AutoThrottle -- Dynamically adjusts HTTP throughput during the scan for maximum bandwidth utilization
        - TimeoutNotice -- Provides a notice for issues uncovered by timing attacks when the affected audited pages returned unusually high response times to begin with.</br>
@@ -188,7 +190,16 @@ Still, this can be an invaluable asset to Fuzzer modules.
 
 The Web User Interface is basically a Sinatra app which acts as an Arachni XMLRPC client and connects to a running XMLRPC Dispatch server.
 
-Thus, you first need to start a Dispatcher like so:
+#### Autostart
+
+There's an autostart script to start all systems that are required by the WebUI:
+    $ arachni_web_autostart
+
+**Note:**: _The "arachni_xmlrpcd" and "arachni_web" executables will need to be in your PATH._
+
+#### Manually
+
+You first need to start a Dispatcher like so:
     $ arachni_xmlrpcd &
 
 Then start the WebUI by running:
@@ -308,13 +319,22 @@ _If you installed the Gem then you'll have to look for the "profiles" directory
 
 ## Installation
 
+### CDE packages for Linux
+
+Arachni is released as [CDE packages](http://stanford.edu/~pgbovine/cde.html) for your convinience.<br/>
+CDE packages are self contained and thus alleviate the need for Ruby and other dependencies to be installed or root access.<br/>
+You can download the latest CDE package from the [download](https://github.com/Zapotek/arachni/downloads) page and escape the dependency hell.<br/>
+If you decide to go the CDE route you can skip the rest, you're done.
+
+
+### Gem
+
 To install the Gem or work with the source code you'll also need the following system libraries:
     $ sudo apt-get install libxml2-dev libxslt1-dev libcurl4-openssl-dev libsqlite3-dev
 
 You will also need to have Ruby 1.9.2 installed *including* the dev package/headers.<br/>
 The prefered ways to accomplish this is by either using [RVM](http://rvm.beginrescueend.com/) or by downloading and compiling the source code for [Ruby 1.9.2](http://www.ruby-lang.org/en/downloads/) manually.
 
-### Gem
 
 To install Arachni:
     $ gem install arachni

data/Rakefile

@@ -56,6 +56,7 @@ task :clean do
     sh "rm *.afr || true"
     sh "rm logs/XMLRPC* || true"
     sh "rm lib/ui/web/server/db/log.db || true"
+    sh "rm lib/ui/web/server/db/default.db || true"
     sh "rm lib/ui/web/server/db/welcomed || true"
 end
 

data/bin/arachni_web_autostart

@@ -0,0 +1,46 @@
+#!/usr/bin/env ruby
+
+=begin
+                  Arachni
+  Copyright (c) 2010-2011 Tasos "Zapotek" Laskos <tasos.laskos@gmail.com>
+
+  This is free software; you can copy and distribute and modify
+  this program under the term of the GPL v2.0 License
+  (See LICENSE file for details)
+
+
+  Simple shell script to start all systems required for the WebUI.
+  Requires all arachni executables to be in PATH.
+
+    Original by: Brandon Potter <bpotter8705@gmail.com>
+    Modified by: Tasos Laskos <tasos.laskos@gmail.com>
+
+=end
+
+# the gemspec doesn't seem to able to handle shell scripts
+# so we hack around it
+exec  <<END
+
+export xterm="xterm -geometry 80X10 -hold"
+
+echo "[>] Starting the Arachni Dispatch server..."
+xterm -T "Arachni Dispatch server" -e "arachni_xmlrpcd" &
+sleep 5
+
+echo "[>] Starting the Arachni WebUI server..."
+xterm -T "Arachni WebUI server" -e "arachni_web" &
+sleep 3
+
+echo "[>] Opening browser..."
+xdg-open http://127.0.0.1:4567
+
+echo "[>] Hit Ctrl+C to shut everything down."
+
+while :
+do
+    sleep 1
+done
+
+exit
+
+END

data/lib/anemone/page.rb

@@ -9,6 +9,7 @@
 =end
 
 require 'nokogiri'
+require Arachni::Options.instance.dir['lib'] + 'nokogiri/xml/node'
 require 'ostruct'
 require 'webrick/cookie'
 

data/lib/arachni.rb

@@ -11,6 +11,6 @@
 module Arachni
 
     # the universal system version
-    VERSION      = '0.2.2.1'
+    VERSION      = '0.2.2.2'
 
 end

data/lib/framework.rb

@@ -57,7 +57,7 @@ module Arachni
 # @author: Tasos "Zapotek" Laskos
 #                                      <tasos.laskos@gmail.com>
 #                                      <zapotek@segfault.gr>
-# @version: 0.2.1
+# @version: 0.2.2
 #
 class Framework
 
@@ -68,6 +68,7 @@ class Framework
     #
     include Arachni::UI::Output
     include Arachni::Module::Utilities
+    include Arachni::Mixins::Observable
 
     # the version of *this* class
     REVISION     = '0.2.1'
@@ -321,6 +322,8 @@ class Framework
             end
         }
 
+        return if @plugin_store[name]
+
         @plugin_store[name] = {
             :results => obj
         }.merge( plugin.class.info )
@@ -453,7 +456,7 @@ class Framework
 
     private
 
-    def clean_up!
+    def clean_up!( skip_audit_queue = false )
         @opts.finish_datetime = Time.now
         @opts.delta_time = @opts.finish_datetime - @opts.start_datetime
 
@@ -466,7 +469,7 @@ class Framework
         @plugins.block!
 
         # a plug-in may have updated the page queue, rock it!
-        audit_queue
+        audit_queue if !skip_audit_queue
 
         # refresh the audit store
         audit_store( true )
@@ -531,6 +534,8 @@ class Framework
     def run_mods( page )
         return if !page
 
+        call_on_run_mods( page.deep_clone )
+
         @current_url = page.url.to_s
 
         @modules.each_pair {

data/lib/http.rb

@@ -16,6 +16,7 @@ require Options.instance.dir['lib'] + 'typhoeus/request'
 require Options.instance.dir['lib'] + 'typhoeus/response'
 require Options.instance.dir['lib'] + 'module/utilities'
 require Options.instance.dir['lib'] + 'module/trainer'
+require Options.instance.dir['lib'] + 'mixins/observable'
 
 #
 # Arachni::Module::HTTP class
@@ -33,13 +34,14 @@ require Options.instance.dir['lib'] + 'module/trainer'
 # @author: Tasos "Zapotek" Laskos
 #                                      <tasos.laskos@gmail.com>
 #                                      <zapotek@segfault.gr>
-# @version: 0.2.3
+# @version: 0.2.5
 #
 class HTTP
 
     include Arachni::UI::Output
     include Singleton
     include Arachni::Module::Utilities
+    include Arachni::Mixins::Observable
 
     #
     # @return [URI]
@@ -86,7 +88,6 @@ class HTTP
 
         hydra_opts = {
             :max_concurrency               => req_limit,
-            :disable_ssl_peer_verification => true,
             :username                      => opts.url.user,
             :password                      => opts.url.password,
             :method                        => :auto,
@@ -125,6 +126,7 @@ class HTTP
         @opts = {
             :user_agent      => opts.user_agent,
             :follow_location => false,
+            :disable_ssl_peer_verification => true,
             # :timeout         => 8000
         }.merge( proxy_opts )
 
@@ -137,11 +139,7 @@ class HTTP
         @curr_res_time = 0
         @curr_res_cnt  = 0
 
-        @on_complete = []
-        @on_queue    = []
-
         @after_run = []
-        @after_run_persistent = []
     end
 
     #
@@ -158,13 +156,9 @@ class HTTP
                 |block|
                 block.call
             }
-
             @after_run.clear
 
-            @after_run_persistent.each {
-                |block|
-                block.call
-            }
+            call_after_run_persistent( )
 
             @curr_res_time = 0
             @curr_res_cnt  = 0
@@ -207,10 +201,7 @@ class HTTP
 
         req.id = @request_count
 
-        @on_queue.each {
-            |block|
-            exception_jail{ block.call( req, async ) }
-        }
+        call_on_queue( req, async )
 
         if( !async )
             @hydra_sync.queue( req )
@@ -237,10 +228,7 @@ class HTTP
             @curr_res_cnt   += 1
             @curr_res_time  += res.start_transfer_time
 
-            @on_complete.each {
-                |block|
-                exception_jail{ block.call( res ) }
-            }
+            call_on_complete( res )
 
             parse_and_set_cookies( res )
 
@@ -280,26 +268,6 @@ class HTTP
         @after_run << block
     end
 
-    def after_run_persistent( &block )
-        @after_run_persistent << block
-    end
-
-    #
-    # Gets called each time a request completes and passes the response
-    # to the block
-    #
-    def on_complete( &block )
-        @on_complete << block
-    end
-
-    #
-    # Gets called each time a request is queued and passes the request
-    # to the block
-    #
-    def on_queue( &block )
-        @on_queue << block
-    end
-
     #
     # Makes a generic request
     #
@@ -656,6 +624,8 @@ class HTTP
         # update framework cookies
         Arachni::Options.instance.cookies = cookie_hash
 
+        call_on_new_cookies( cookie_hash, res )
+
         current = parse_cookie_str( @init_headers['cookie'] )
         set_cookies( current.merge( cookie_hash ) )
     end

data/lib/mixins/observable.rb

@@ -0,0 +1,87 @@
+=begin
+                  Arachni
+  Copyright (c) 2010-2011 Tasos "Zapotek" Laskos <tasos.laskos@gmail.com>
+
+  This is free software; you can copy and distribute and modify
+  this program under the term of the GPL v2.0 License
+  (See LICENSE file for details)
+
+=end
+
+module Arachni
+
+require Options.instance.dir['lib'] + 'module/utilities'
+
+module Mixins
+
+#
+# Provides a flexible way to make any Class observable via callbacks/hooks
+# using simple dynamic programming with the help of "method_missing()".
+#
+# The observable classes (those which include this module) use:
+#    * call_<hookname>( *args )
+# to call specific hooks.
+#
+# The observers set hooks using:
+#    * observer_instance.add_<hookname>( &block )
+#
+#
+# @author: Tasos "Zapotek" Laskos
+#                                      <tasos.laskos@gmail.com>
+#                                      <zapotek@segfault.gr>
+# @version: 0.1
+#
+module Observable
+
+    include Arachni::Module::Utilities
+
+    def method_missing( sym, *args, &block )
+
+        # grab the action (add/call) and the hook name
+        action, hook = sym.to_s.split( '_', 2 )
+
+        @__hooks       ||= {}
+        @__hooks[hook] ||= []
+
+        if( action && hook )
+            case action
+
+            when 'add'
+                add_block( hook, &block )
+                return
+
+             when 'call'
+                call_blocks( hook, args )
+                return
+            end
+        end
+
+        raise NoMethodError.new( "Undefined method '#{sym.to_s}'.", sym, args )
+    end
+
+    private
+
+    def add_block( hook, &block )
+        @__hooks[hook] << block
+    end
+
+    def call_blocks( hook, *args )
+        @__hooks[hook].each {
+            |block|
+
+            exception_jail {
+
+                if args.flatten.size == 1
+                    block.call( args.flatten[0] )
+                else
+                    block.call( *args )
+                end
+            }
+
+        }
+    end
+
+end
+
+end
+end