arachni 0.4.5.2 → 0.4.6

checksums.yaml

@@ -1,15 +1,15 @@
 ---
 !binary "U0hBMQ==":
   metadata.gz: !binary |-
-    NzNlOTZlMGUyOTkwNTY4MWRmYTQyYmFmNmRlMGI0ZDVlNzZkMmI4NA==
+    MjQyMTU0MWMwYTcyZTVhMjk0NDM0YjZlMmZiMDllYmI4ZTNkZjM4Zg==
   data.tar.gz: !binary |-
-    YmNiN2YyNmFlMzZlZThjYmZiMGUwZGY1MTQ5NjlhZTgyYzk4M2NjZQ==
-!binary "U0hBNTEy":
+    MWI4ZjZmNzE5MzMxM2UyOTdkMzI4NjM0YTI0NWQwZDg4NmUxMzcwYw==
+SHA512:
   metadata.gz: !binary |-
-    MDZmZGMzZTU5ZjVhM2Q4MDg2OTI5NGUwMDBhZTJhNGRlOGViYjc4YjlmY2I3
-    OWE4Y2E4MTFkODIyNTI0N2VmNzBmM2Q2YjEwZGQxMmZlNjFjODQ3YTFmYmZh
-    N2Y5NDhhNzgyMTQzZmM0MTRiNWMzNmYwZWEwOTVjY2ExZWQ3NjA=
+    N2NkYzk3M2Y1MzY3YWQ0ZmEwZTM0ODBiZGUwNmZhYmIxOWRlODAyZDFjOTgw
+    YTIwMmQ0YjdhNDhjMWUzNWVkYzQ0MTM0ZjA5ZGQyYzM2MzQ2MDgyZWFiZmQw
+    MDY5NDFiMWY0ZGY4YzNjMWZjNTliMmFlM2YyZGM2MzFlZjY4YTg=
   data.tar.gz: !binary |-
-    MzRiNTI1ZDY0Y2EwZjBlNmRiYWRjOGE0ZmI5ODkyMmQyZjMzN2VlNWVkZTg3
-    ZTkzMDEzMjBlZGZiMGM2MmI5MTkyZmE5ZmRiZTY0NzcwMDc2YzY3M2JkN2Y4
-    ZmY0MDhiMTAyMDU5YWRlZGEwMjMzOGRjNjhiMTE2NzcxNDhjZjU=
+    M2JlNDdkN2MxODRhMThkYzQxNTBkZjcyMWQxNTg3ZWM2YzU5NTQyOTRhNmFm
+    ZTVkNzA4MjJjNmI2NzQ0ZTE2ZjNkNzgzNmFlMGJkYWQwMGRhNjc0M2RiODM5
+    YjY3MGM1MGRiNWFmYjI0MTg2YjVmNjZjYzhiOGZhNTk3NGM4YTE=

data/CHANGELOG.md

@@ -1,5 +1,102 @@
 # ChangeLog
 
+## 0.4.6 _(January 1, 2014)_
+
+- CLI user interfaces
+    - `--lsmod`
+        - Longer pauses every 3 modules, it lists all of them at once.
+        - Updated to show the _Severity_ of the issues the module logs.
+    - `Ctrl+C` screen optimized to use less resources when printing scan data.
+- Options
+    - `--cookie-string` -- Updated to also handle cookies in the form of `Set-Cookie` headers.
+    - Added:
+        - `--external-address` -- The external address of a Dispatcher.
+        - `--http-queue-size` -- Maximum amount of requests to keep in the queue,
+            bigger size means better scheduling and better performance, smaller
+            means less RAM consumption.
+- `Session`
+    - `#ensure_logged_in` -- Retry on login failure.
+- `Spider`
+    - Don't apply scope restrictions to the seed URL.
+- `Framework`
+    - Audit
+        - Stored pages are now offloaded to disk to lower RAM consumption.
+- `Trainer`
+    - `#push` -- Prints verbose messages in cases of scope violations.
+- `HTTP`
+    - Maximum request-queue size lowered from 5000 to 500, to decrease RAM usage
+        by preventing the storage of large amounts of requests for extended periods of time.
+    - Updated to use the new `Support::Signature` class for custom-404 signatures.
+- `RPC::Server::Dispatcher`
+    - Now supports specifying an external address to allow for deployments behind NATs.
+- `Element::Capabilities::Auditable::RDiff`
+    - Updated to use the new `Support::Signature` class to perform response body comparisons.
+    - Updated the algorithm to use a `false` as the control.
+    - Added integrity check for the analysis process.
+    - Optimized scheduling of data gathering.
+    - Reduced total amount of performed requests.
+    - Massively reduced RAM consumption for data storage and analysis.
+- `Element::Capabilities::Auditable::Timeout`
+    - Updated the algorithm to use an approximated web application processing
+        time instead of the HTTP timeout based on the total request-response process.
+    - Made analysis corruption checks more stringent to diminish the chances of
+        false positives.
+    - Fixed bug causing non-vetted inputs to reach the final stages of analysis
+        which sometimes resulted in false positives.
+    - Added a cool-off period after Phase 2 to ensure webapp responsiveness post-attack.
+    - Improved status messages.
+- `Element::Capabilities::Auditable::Taint`
+    - Added longest-word-optimization -- Checks if the longest word of a regexp
+        exists in the response body prior to matching the full-blown regexp.
+- `Element::Capabilities::Auditable#audit`
+    - Added option `:skip_like`, accepting blocks used to filter the mutations
+        about to be audited.
+    - Fixed bug causing audits with constantly changing tokens to fail.
+    - Updated to use `#each_mutation` instead of `#mutations`.
+- `Element::Capabilities::Mutable`
+    - Added `#each_mutation` to generate mutations on the fly instead of relying
+        on `#mutations` to generate an array of mutations.
+    - Updated `#mutations` to delegate to `#each_mutation`.
+- `Element::Cookie#encode`
+    - Allow `=` to remain un-encoded in the cookie value.
+- `Element::Form` -- Buttons are now treated as inputs as well.
+- `Options#load` -- Updated to support serialized `Hash` objects.
+- Added `Support::Signature` -- Signature class used to generate and refine signatures
+    from `String` objects.
+- Modules
+    - Audit
+        - `path_traversal` -- Updated to use double-slashes for *nix payloads.
+        - `file_inclusion` -- Added evasive payloads using '\'.
+        - `source_code_disclosure`
+            - Increased coverage by following the directory tree of each file one
+                level at a time.
+        - `xss_script_tag` -- Updated to check for the existence of encoding operations.
+        - `sqli`
+            - Updated to cache the compiled regular expressions.
+            - Updated to use the longest-word-optimization of the taint analysis
+                implementation for faster analysis.
+        - `sqli_blind_rdiff`
+            - Massively reduced injected payloads.
+        - `os_cmd_injection_timing` -- Decreased the time delay.
+    - Recon
+        - `localstart_asp`
+            - Check for an ASP platform instead of a Windows one.
+            - Fixed `LocalJumpError`.
+- Plugins
+    - `autologin`
+        - Changed `print_bad` to `print_error` so that errors are written to the
+            error log.
+        - Scan remains paused and awaits user action upon failure.
+    - `proxy`
+        - Updated request URL encoding to handle malformed URLs.
+        - Disabled reverse DNS lookup on requests to increase performance.
+    - `content_types` -- Moved out of `defaults/'.
+    - `cookie_collector`
+        - Added `filter` option used to determine which cookies to log based on
+            a pattern matched against cookie names.
+- Reports -- Added `content_type` to all reports with `outfile` option in `.info`.
+    - `xml` -- Escaped parameter values in XML report.
+
 ## 0.4.5.2 _(September 18, 2013)_
 
 - `gemspec`

data/CONTRIBUTORS.md

@@ -15,6 +15,7 @@ suggestions or testing it.
 - [Evan Beard](mailto:beard.evan@gmail.com) for feedback and patches.
 - [Michael Borohovski](mailto:borski@mit.edu) for testing, feedback and patches.
 - [Ben Sedat](mailto:bsedat@alum.mit.edu) for testing, feedback and patches.
+- [Michiel van Es](mailto:mve@pragmasec.nl) for relentless testing and feedback.
 
 A big thanks to my buddy [Andreas](mailto:rainmakergr@gmail.com) for the original
 spider drawing used in the project graphics.

data/NOTICE

@@ -1,5 +1,5 @@
     Arachni Web Application Security Scanner Framework
-    Copyright 2010-2013 Tasos Laskos <tasos.laskos@gmail.com>
+    Copyright 2010-2014 Tasos Laskos <tasos.laskos@gmail.com>
 
     This product includes code from the Arachni Web Application Security Scanner Framework
     developed by Tasos Laskos <tasos.laskos@gmail.com>.

data/README.md

@@ -3,7 +3,7 @@
 <table>
     <tr>
         <th>Version</th>
-        <td>0.4.5.2</td>
+        <td>0.4.6</td>
     </tr>
     <tr>
         <th>Homepage</th>
@@ -38,7 +38,7 @@
     </tr>
     <tr>
         <th>Copyright</th>
-        <td>2010-2013 Tasos Laskos</td>
+        <td>2010-2014 Tasos Laskos</td>
     </tr>
     <tr>
         <th>License</th>
@@ -403,6 +403,8 @@ core remains lean and makes it easy for anyone to add arbitrary functionality.
 - Script (`script`) -- Loads and runs an external Ruby script under the scope of a plugin,
     used for debugging and general hackery.
 - Uncommon headers (`uncommon_headers`) -- Logs uncommon headers.
+- Content-types (`content_types`) -- Logs content-types of server responses aiding in the
+    identification of interesting (possibly leaked) files.
 
 #### Defaults
 
@@ -410,8 +412,6 @@ Default plugins will run for every scan and are placed under `/plugins/defaults/
 
 - AutoThrottle (`autothrottle`) -- Dynamically adjusts HTTP throughput during the scan for
     maximum bandwidth utilization.
-- Content-types (`content_types`) -- Logs content-types of server responses aiding in the
-    identification of interesting (possibly leaked) files.
 - Healthmap (`healthmap`) -- Generates sitemap showing the health of each crawled/audited URL
 - Resolver (`resolver`) -- Resolves vulnerable hostnames to IP addresses.
 

data/Rakefile

@@ -1,5 +1,5 @@
 =begin
-    Copyright 2010-2013 Tasos Laskos <tasos.laskos@gmail.com>
+    Copyright 2010-2014 Tasos Laskos <tasos.laskos@gmail.com>
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.
@@ -15,6 +15,7 @@
 =end
 
 require 'bundler'
+require 'fileutils'
 require File.expand_path( File.dirname( __FILE__ ) ) + '/lib/arachni'
 
 begin
@@ -22,27 +23,116 @@ begin
     require 'rspec/core/rake_task'
 
     namespace :spec do
+
+        desc 'Run core library tests.'
         RSpec::Core::RakeTask.new( :core ) do |t|
-            t.pattern = FileList[ "spec/arachni/**/*_spec.rb" ]
+            t.pattern = FileList[ 'spec/arachni/**/*_spec.rb' ]
         end
 
+        desc 'Run module tests.'
         RSpec::Core::RakeTask.new( :modules ) do |t|
-            t.pattern = FileList[ "spec/modules/**/*_spec.rb" ]
+            t.pattern = FileList[ 'spec/modules/**/*_spec.rb' ]
+        end
+
+        namespace :modules do
+            desc 'Run tests for the audit modules.'
+            RSpec::Core::RakeTask.new( :audit ) do |t|
+                t.pattern = FileList[ 'spec/modules/audit/**/*_spec.rb' ]
+            end
+
+            desc 'Run tests for the recon modules.'
+            RSpec::Core::RakeTask.new( :recon ) do |t|
+                t.pattern = FileList[ 'spec/modules/recon/**/*_spec.rb' ]
+            end
         end
 
+        desc 'Run report tests.'
         RSpec::Core::RakeTask.new( :reports ) do |t|
-            t.pattern = FileList[ "spec/reports/**/*_spec.rb" ]
+            t.pattern = FileList[ 'spec/reports/**/*_spec.rb' ]
         end
 
+        desc 'Run plugin tests.'
         RSpec::Core::RakeTask.new( :plugins ) do |t|
-            t.pattern = FileList[ "spec/plugins/**/*_spec.rb" ]
+            t.pattern = FileList[ 'spec/plugins/**/*_spec.rb' ]
         end
 
+        desc 'Run path-extractor tests.'
         RSpec::Core::RakeTask.new( :path_extractors ) do |t|
-            t.pattern = FileList[ "spec/path_extractors/**/*_spec.rb" ]
+            t.pattern = FileList[ 'spec/path_extractors/**/*_spec.rb' ]
+        end
+
+        desc 'Run external test suites.'
+        RSpec::Core::RakeTask.new( :external ) do |t|
+            t.pattern = FileList[ 'spec/external/**/*_spec.rb' ]
+        end
+
+        namespace :external do
+
+            desc 'Run the WAVSEP test suite.'
+            RSpec::Core::RakeTask.new( :wavsep ) do |t|
+                t.pattern = FileList[ 'spec/external/wavsep/**/**/*_spec.rb' ]
+            end
+
+            namespace :wavsep do
+
+                desc 'Run the WAVSEP active tests.'
+                RSpec::Core::RakeTask.new( :active ) do |t|
+                    t.pattern = FileList[ 'spec/external/wavsep/active/**/*_spec.rb' ]
+                end
+
+                namespace :active do
+
+                    desc 'Run the WAVSEP XSS tests.'
+                    RSpec::Core::RakeTask.new( :xss ) do |t|
+                        t.pattern = FileList[ 'spec/external/wavsep/active/xss_spec.rb' ]
+                    end
+
+                    desc 'Run the WAVSEP SQL injection tests.'
+                    RSpec::Core::RakeTask.new( :sqli ) do |t|
+                        t.pattern = FileList[ 'spec/external/wavsep/active/sqli_spec.rb' ]
+                    end
+
+                    desc 'Run the WAVSEP LFI tests.'
+                    RSpec::Core::RakeTask.new( :lfi ) do |t|
+                        t.pattern = FileList[ 'spec/external/wavsep/active/lfi_spec.rb' ]
+                    end
+
+                    desc 'Run the WAVSEP RFI tests.'
+                    RSpec::Core::RakeTask.new( :rfi ) do |t|
+                        t.pattern = FileList[ 'spec/external/wavsep/active/rfi_spec.rb' ]
+                    end
+                end
+
+                desc 'Run the WAVSEP false positive tests.'
+                RSpec::Core::RakeTask.new( :false_positives ) do |t|
+                    t.pattern = FileList[ 'spec/external/wavsep/false_positives/**/*_spec.rb' ]
+                end
+
+                namespace :false_positives do
+                    desc 'Run the WAVSEP XSS false positive tests.'
+                    RSpec::Core::RakeTask.new( :xss ) do |t|
+                        t.pattern = FileList[ 'spec/external/wavsep/false_positives/xss_spec.rb' ]
+                    end
+
+                    desc 'Run the WAVSEP SQL injection false positive tests.'
+                    RSpec::Core::RakeTask.new( :sqli ) do |t|
+                        t.pattern = FileList[ 'spec/external/wavsep/false_positives/sqli_spec.rb' ]
+                    end
+
+                    desc 'Run the WAVSEP LFI false positive tests.'
+                    RSpec::Core::RakeTask.new( :lfi ) do |t|
+                        t.pattern = FileList[ 'spec/external/wavsep/false_positives/lfi_spec.rb' ]
+                    end
+
+                    desc 'Run the WAVSEP RFI false positive tests.'
+                    RSpec::Core::RakeTask.new( :rfi ) do |t|
+                        t.pattern = FileList[ 'spec/external/wavsep/false_positives/rfi_spec.rb' ]
+                    end
+                end
+            end
         end
 
-        desc "Generate an AFR report for the report tests"
+        desc 'Generate an AFR report for the report tests.'
         namespace :generate do
             task :afr do
 
@@ -78,7 +168,7 @@ begin
                 Arachni::Framework.new.modules.load_all
 
                 Arachni::AuditStore.new( issues: issues.uniq ).
-                    save( 'spec/fixtures/auditstore.afr' )
+                    save( 'spec/support/fixtures/auditstore.afr' )
 
                 Arachni::Options.reset
             end
@@ -91,7 +181,7 @@ rescue LoadError
     puts '  gem install rspec'
 end
 
-desc "Generate docs"
+desc 'Generate docs.'
 task :docs do
 
     outdir = "../arachni-docs"
@@ -103,7 +193,7 @@ task :docs do
     sh "rm -rf .yardoc"
 end
 
-desc "Generate graphics"
+desc 'Generate graphics.'
 task :gfx do
 
     outdir = 'gfx/compiled'
@@ -129,7 +219,7 @@ end
 #
 # [1] https://github.com/tmm1/perftools.rb
 #
-desc "Profile Arachni"
+desc 'Profile Arachni.'
 task :profile do
 
     if !Gem::Specification.find_all_by_name( 'perftools.rb' ).empty?
@@ -144,27 +234,22 @@ task :profile do
 
 end
 
-#
-# Cleans reports and logs
-#
-desc "Cleaning report and log files."
+desc 'Remove report and log files.'
 task :clean do
+    files = %w(error.log *.afr *.yaml *.json *.marshal *.gem pkg/*.gem logs/*.log
+        spec/support/logs/*.log).map { |file| Dir.glob( file ) }.flatten
 
-    sh "rm error.log || true"
-    sh "rm *.afr || true"
-    sh "rm *.yaml || true"
-    sh "rm *.json || true"
-    sh "rm *.marshal || true"
-    sh "rm *.gem || true"
-    sh "rm logs/*.log || true"
-    sh "rm spec/support/logs/*.log || true"
-end
+    next if files.empty?
 
+    puts 'Removing:'
+    files.each { |file| puts "  * #{file}" }
+    FileUtils.rm files
+end
 
 Bundler::GemHelper.install_tasks
 
-desc "Push a new version to RubyGems"
+desc 'Push a new version to RubyGems'
 task :publish => [ :release ]
 
-desc "Build Arachni and run all the tests."
+desc 'Build Arachni and run all the tests.'
 task :default => [ :build, :spec ]

data/arachni.gemspec

@@ -1,6 +1,6 @@
 # coding: utf-8
 =begin
-    Copyright 2010-2013 Tasos Laskos <tasos.laskos@gmail.com>
+    Copyright 2010-2014 Tasos Laskos <tasos.laskos@gmail.com>
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.
@@ -114,7 +114,7 @@ GitHub page        - http://github.com/Arachni/arachni
 Code Documentation - http://rubydoc.info/github/Arachni/arachni
 Author             - Tasos "Zapotek" Laskos (http://twitter.com/Zap0tek)
 Twitter            - http://twitter.com/ArachniScanner
-Copyright          - 2010-2013 Tasos Laskos
+Copyright          - 2010-2014 Tasos Laskos
 License            - Apache License v2
 
 Please do not hesitate to ask for assistance (via the support portal)

data/bin/arachni

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2013 Tasos Laskos <tasos.laskos@gmail.com>
+    Copyright 2010-2014 Tasos Laskos <tasos.laskos@gmail.com>
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.

data/bin/arachni_console

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2013 Tasos Laskos <tasos.laskos@gmail.com>
+    Copyright 2010-2014 Tasos Laskos <tasos.laskos@gmail.com>
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.

data/bin/arachni_multi

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2013 Tasos Laskos <tasos.laskos@gmail.com>
+    Copyright 2010-2014 Tasos Laskos <tasos.laskos@gmail.com>
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.

data/bin/arachni_rpc

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2013 Tasos Laskos <tasos.laskos@gmail.com>
+    Copyright 2010-2014 Tasos Laskos <tasos.laskos@gmail.com>
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.

data/bin/arachni_rpcd

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2013 Tasos Laskos <tasos.laskos@gmail.com>
+    Copyright 2010-2014 Tasos Laskos <tasos.laskos@gmail.com>
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.