arachni 1.4 → 1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f4c2f78464615f487fac68b9e72685088d568977
-  data.tar.gz: db89773e54c4321dbb18bd9c18958c3263f4c952
+  metadata.gz: 9258760c9e75d398da9ab655640904e5d51544c4
+  data.tar.gz: b7f5f56e8f1977916b7a87465954c354892cc4d3
 SHA512:
-  metadata.gz: 5ed261c2eab8f545585e8ac2a733f59296823e8d3de533a6789ba871c4a4ff732108760c33169198fde6b8c240192cdbeba16a9241d8eba4a03da6b9ee9b176b
-  data.tar.gz: 50297d450fd4e8d53b1c556af729d889a94ea0c4cd2f7220e2703dd70f274bff373d6603fb57dfb99860d3fd90199b0258d0a6e271d672e92564884cf8be5a1f
+  metadata.gz: baef5531633d799813d4bbfdfc89a44aa682feb5b502349e40c19ec9036ef074f2cf4f8838403487e79b2ac905ace671863b5ee092515251418d0180f0db659b
+  data.tar.gz: 43e379175dff426bc078a5055dda4b0d65dee05f47df7032559501ce1603d5e4c32517ff0c9fb69669a0d21b7710df4f62b2899de099e32ebcc530165a039c46

data/CHANGELOG.md

@@ -1,5 +1,140 @@
 # ChangeLog
 
+## 1.5 _(January 31, 2017)_
+
+- Executables
+    - `arachni_rpcd_monitor` -- Brought up to date with Dispatcher refactoring.
+    - New
+        - `arachni_reproduce` -- Reproduces the issues in the given report.
+- Options
+    - `url` -- Raise error on addresses starting with `127.` because
+        PhantomJS 2.1.1 doesn't proxy any loopback connections.
+    - `--http-cookie-string` -- Updated to only accept `Set-Cookie` formatted
+        cookies instead of `Cookie` ones.
+    - `--browser-cluster-job-timeout`
+        - Repurposed to apply to communication requests for Selenium rather than
+            the entire job.
+        - Lowered to `10` seconds.
+    - New
+        - `--http-authentication-type`
+            - `auto` -- Default
+            - `basic`
+            - `digest`
+            - `digest_ie`
+            - `negotiate`
+            - `ntlm`
+        - `--scope-dom-event-limit` -- Limits the amount of DOM events to be
+            triggered for each DOM depth.
+        - `--daemon-friendly` -- Disables status screen.
+- `UI`
+    - `CLI`
+        - `Framework` -- Trap `USR1` signal and go into a `pry` session for debugging.
+- `URI`
+    - `.fast_parse` --- Ignore `data:` URIs.
+- `HTTP`
+    - `ProxyServer`
+        - Fixed state of abruptly closed SSL interceptor connections leading to
+            frozen browser operations.
+        - Added support for configurable concurrency of origin requests to keep
+            the amount of `Thread`s low.
+        - Added support for `Connection: Upgrade` requests by tunneling WebSocket
+            connections.
+    - `Client`
+        - Added `X-Arachni-Scan-Seed` header that includes the random scan seed.
+        - `Dynamic404Handler`
+            - Added more training scenarios for when:
+                - Dashes are used as routing separators.
+                - Directory name prepending and appending is ignored.
+            - Updated to not dismiss redirects but follow the location.
+- `Browser`
+    - Updated engine to PhantomJS 2.1.1.
+    - Remove `Content-Security-Policy` to allow the Arachni JS env to run.
+    - `#snapshot_id` -- Moved to browser-side `DOMMonitor` for better performance.
+    - `#capture` -- Extract query parameters from `POST` requests.
+    - `#capture_snapshot` -- Deduplicate based on DOM URL and transitions as well.
+    - `ElementLocator` -- Fixed bug causing broken CSS selectors with UTF8 characters.
+    - `Javascript`
+        - `#dom_elements_with_events`
+            - Moved code to browser-side `DOMMonitor`.
+            - Updated it to return results in batches, in order to keep RAM
+                usage under control when processing large pages with thousands
+                of elements with events.
+- `BrowserCluster`
+    - `Worker`
+        - `#run_job` -- Retry 5 times on job time-outs.
+- `Element`
+    - `Capabilities`
+        - `Auditable`
+            - New
+                - `Buffered` -- Reads audit responses in chunks.
+                - `LineBuffered` -- Reads audit responses in chunks of lines.
+    - `DOM`
+        - `Capabilities`
+            - `Submittable`, `Auditable` -- Switched from `Proc` to class methods
+                for callbacks, in order to avoid keeping contexts in memory.
+- Session -- Allow for a submit input to be specified when the login needs to be
+    triggered by clicking it, rather than just triggering the submit event on
+    the form.
+- REST API
+    - Added `GET /scans/:id/summary` to return scan progress data without
+        `issues`, `errors` and `sitemap`.
+- Report
+    - Added `#seed` attribute that includes the random scan seed.
+- Plugins
+    - New
+        - `webhook_notify` -- Sends a webhook payload over HTTP at the end of the scan.
+        - `rate_limiter` -- Rate limits HTTP requests.
+        - `page_dump` -- Dumps page data to disk as YAML.
+    - `proxy` -- `bind_address` default switched to `127.0.0.1`, `0.0.0.0` breaks
+        SSL interception on MS Windows.
+    - `metrics`
+        - Fixed division by 0 error when no requests have been performed.
+        - Added:
+            - HTTP
+                - Request time-outs
+                - Responses per second
+            - Browser cluster
+                - Timed-out jobs
+                - Seconds per job
+                - Total job time
+                - Job count
+    - `email_notify`
+        - Retry on error.
+        - Default to `afr` as a report format.
+- Checks
+    - Active
+        - `xss` -- Only check HTML responses to avoid FPs.
+        - `xss_event`
+            - Replaced full parsing of responses with SAX.
+            - Only check HTML responses to avoid FPs.
+        - `xss_script_context`
+            - Replaced full parsing of responses with SAX.
+            - Only check HTML responses to avoid FPs.
+        - `xss_tag`
+            - Replaced full parsing of responses with SAX.
+            - Only check HTML responses to avoid FPs.
+        - `unvalidated_redirect`, `unvalidated_redirect_dom`, `xss`, `xss_dom`,
+            `xss_dom_script_context`, `xss_script_context` -- Replaced `Proc`s
+                with class methods for `BrowserCluster` job callbacks.
+        - `unvalidated_redirect` -- Added prepended payload to the default value.
+        - `sql_injection` -- Added more error signatures for HSQLDB, Java and SQLite.
+        - `csrf` -- Removed heuristics that try to match tokens based on format;
+            now only uses a nonce check.
+        - `path_traversal` -- Increased maximum traversals to 8.
+    - Passive
+        - `backup_files`
+            - Ignore media files to avoid FPs when dealing with galleries and the like.
+            - Added issue remark explaining how the original resource name was manipulated.
+        - `backup_directories` -- Added issue remark explaining how the original
+            resource name was manipulated.
+        - `xst` -- Run once for each protocol, not just for the first page.
+- Path extractors
+    - `data_url` -- Extract from all elements, not just links.
+- Reporters
+    - `xml`
+        - Replaced unsupported null-bytes with a placeholder.
+        - Made `issues/issue/page/dom/data_flow_sinks/data_flow_sink/frame/line` nil-able.
+
 ## 1.4 _(February 7, 2016)_
 
 - Native MS Windows compatibility.
@@ -95,6 +230,7 @@
                 there's no way to verify SSNs.
             - `http_only_cookies`, `insecure_cookies` -- Only check current page
                 cookies, don't let the CookieJar ones sneak in.
+            - `insecure_cookies` -- Check JS cookies too.
 - Plugins
     - `proxy`
         - Removed injection of control toolbar to each response.

data/Gemfile

@@ -1,6 +1,7 @@
 source 'https://rubygems.org'
 
-gem 'rake'
+gem 'rake', '11.3.0'
+gem 'pry'
 
 group :docs do
     gem 'yard'
@@ -19,6 +20,7 @@ group :prof do
     gem 'sys-proctable'
     gem 'ruby-mass'
     gem 'benchmark-ips'
+    gem 'memory_profiler'
 end
 
 gemspec

data/LICENSE.md

@@ -1,6 +1,6 @@
 # License
 
-Copyright 2010-2016 [Tasos Laskos](mailto:tasos.laskos@arachni-scanner.com).
+Copyright 2010-2017 [Sarosys LLC](http://www.sarosys.com).
 
 ```
                     Arachni Public Source License

data/README.md

@@ -3,7 +3,7 @@
 <table>
     <tr>
         <th>Version</th>
-        <td>1.4</td>
+        <td>1.5</td>
     </tr>
     <tr>
         <th>Homepage</th>
@@ -38,7 +38,7 @@
     </tr>
     <tr>
         <th>Copyright</th>
-        <td>2010-2016 Tasos Laskos</td>
+        <td>2010-2017 <a href="http://www.sarosys.com">Sarosys LLC</a></td>
     </tr>
     <tr>
         <th>License</th>
@@ -555,6 +555,9 @@ core remains lean and makes it easy for anyone to add arbitrary functionality.
 - Metrics (`metrics`) -- Captures metrics about multiple aspects of the scan and the web application.
 - Restrict to DOM state (`restrict_to_dom_state`) -- Restricts the audit to a single page's DOM
     state, based on a URL fragment.
+- Webhook notify (`webhook_notify`) -- Sends a webhook payload over HTTP at the end of the scan.
+- Rate limiter (`rate_limiter`) -- Rate limits HTTP requests.
+- Page dump (`page_dump`) -- Dumps page data to disk as YAML.
 
 ##### Defaults
 

data/Rakefile

@@ -1,5 +1,5 @@
 =begin
-    Copyright 2010-2016 Tasos Laskos <tasos.laskos@arachni-scanner.com>
+    Copyright 2010-2017 Sarosys LLC <http://www.sarosys.com>
 
     This file is part of the Arachni Framework project and is subject to
     redistribution and commercial restrictions. Please see the Arachni Framework

data/arachni.gemspec

@@ -1,6 +1,6 @@
 # coding: utf-8
 =begin
-    Copyright 2010-2016 Tasos Laskos <tasos.laskos@arachni-scanner.com>
+    Copyright 2010-2017 Sarosys LLC <http://www.sarosys.com>
 
     This file is part of the Arachni Framework project and is subject to
     redistribution and commercial restrictions. Please see the Arachni Framework
@@ -10,7 +10,7 @@
 Gem::Specification.new do |s|
     require File.expand_path( File.dirname( __FILE__ ) ) + '/lib/arachni/version'
 
-    s.required_ruby_version = '>= 2.0.0'
+    s.required_ruby_version = '>= 2.2.0'
 
     s.name              = 'arachni'
     s.version           = Arachni::VERSION
@@ -37,77 +37,82 @@ Gem::Specification.new do |s|
 
     s.executables       = Dir.glob( 'bin/*' ).map { |e| File.basename e }
 
-    s.extra_rdoc_files  = %w(README.md ACKNOWLEDGMENTS.md LICENSE.md
-                            AUTHORS.md CHANGELOG.md CONTRIBUTORS.md)
+    s.extra_rdoc_files  = %w(README.md LICENSE.md CHANGELOG.md)
 
     s.rdoc_options      = [ '--charset=UTF-8' ]
 
-    s.add_dependency 'awesome_print'
+    s.add_dependency 'awesome_print',       '1.6.1'
 
-    s.add_dependency 'rack'
+    s.add_dependency 'rack',                '1.6.4'
 
+    # Don't specify version, messes with the packages since they always grab the
+    # latest one.
     s.add_dependency 'bundler'
 
-    s.add_dependency 'concurrent-ruby',     '1.0.0'
-    s.add_dependency 'concurrent-ruby-ext', '1.0.0'
+    s.add_dependency 'concurrent-ruby',     '1.0.2'
+    s.add_dependency 'concurrent-ruby-ext', '1.0.2'
 
     # For compressing/decompressing system state archives.
-    s.add_dependency 'rubyzip',           '1.1.6'
+    s.add_dependency 'rubyzip',             '1.1.6'
 
     # HTTP proxy server
-    s.add_dependency 'http_parser.rb'
+    s.add_dependency 'http_parser.rb',      '0.6.0'
 
     # HTML report
-    s.add_dependency 'coderay',           '1.1.0'
+    s.add_dependency 'coderay',             '1.1.0'
 
-    s.add_dependency 'childprocess',      '0.5.3'
+    s.add_dependency 'childprocess',        '0.5.3'
 
     # RPC serialization.
-    s.add_dependency 'msgpack',           '0.7.0'
+    s.add_dependency 'msgpack',             '0.7.0'
 
     if RUBY_PLATFORM != 'java'
         # Optimized JSON.
-        s.add_dependency 'oj',            '~> 2.14.3'
-        s.add_dependency 'oj_mimic_json'
+        s.add_dependency 'oj',              '2.15.0'
+        s.add_dependency 'oj_mimic_json',   '1.0.1'
     end
 
     # Web server
-    s.add_dependency 'puma',              '2.14.0'
+    s.add_dependency 'puma',                '2.14.0'
 
     # REST API
-    s.add_dependency 'sinatra',           '1.4.6'
-    s.add_dependency 'sinatra-contrib',   '1.4.6'
+    s.add_dependency 'sinatra',             '1.4.6'
+    s.add_dependency 'sinatra-contrib',     '1.4.6'
 
     # RPC client/server implementation.
-    s.add_dependency 'arachni-rpc',       '0.2.1.3'
+    s.add_dependency 'arachni-rpc',         '~> 0.2.1.4'
 
     # HTTP client.
-    s.add_dependency 'typhoeus',          '1.0.1'
+    s.add_dependency 'typhoeus',            '1.0.2'
 
     # Fallback URI parsing and encoding utilities.
-    s.add_dependency 'addressable',       '2.3.6'
+    s.add_dependency 'addressable',         '2.3.6'
 
     # E-mail plugin.
-    s.add_dependency 'pony',              '1.8'
+    s.add_dependency 'pony',                '1.11'
 
     # For the Arachni console (arachni_console).
-    s.add_dependency 'rb-readline',       '0.5.1'
+    s.add_dependency 'rb-readline',         '0.5.1'
 
-    # Markup parsing.
-    s.add_dependency 'nokogiri',          '1.6.8rc2'
+    # Markup parsing, for reports and Element::XML.
+    s.add_dependency 'nokogiri',            '1.6.8.1'
+    # Really fast and lightweight markup parsing, for pages.
+    s.add_dependency 'ox',                  '2.4.9'
 
     # Outputting data in table format (arachni_rpcd_monitor).
-    s.add_dependency 'terminal-table',    '1.4.5'
+    s.add_dependency 'terminal-table',      '1.4.5'
 
     # Browser support for DOM/JS/AJAX analysis stuff.
-    s.add_dependency 'watir-webdriver',   '0.8.0'
+    # Lock webdriver, newer versions has issues.
+    s.add_dependency 'selenium-webdriver',  '3.0.1'
+    s.add_dependency 'watir-webdriver',     '0.8.0'
 
     # Markdown to HTML conversion, used by the HTML report for component
     # descriptions.
-    s.add_dependency 'kramdown',          '1.4.1'
+    s.add_dependency 'kramdown',            '1.4.1'
 
     # Used to scrub Markdown for XSS etc.
-    s.add_dependency 'loofah',            '~> 2.0.0'
+    s.add_dependency 'loofah',              '2.0.3'
 
     s.post_install_message = <<MSG
 
@@ -124,7 +129,7 @@ License            - Arachni Public Source License v1.0
                         (https://github.com/Arachni/arachni/blob/master/LICENSE.md)
 Author             - Tasos "Zapotek" Laskos (http://twitter.com/Zap0tek)
 Twitter            - http://twitter.com/ArachniScanner
-Copyright          - 2010-2016 Tasos Laskos
+Copyright          - 2010-2017 Sarosys LLC (http://www.sarosys.com)
 
 Please do not hesitate to ask for assistance (via the support portal)
 or report a bug (via GitHub Issues) if you come across any problem.

data/bin/arachni

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2016 Tasos Laskos <tasos.laskos@arachni-scanner.com>
+    Copyright 2010-2017 Sarosys LLC <http://www.sarosys.com>
 
     This file is part of the Arachni Framework project and is subject to
     redistribution and commercial restrictions. Please see the Arachni Framework

data/bin/arachni_console

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2016 Tasos Laskos <tasos.laskos@arachni-scanner.com>
+    Copyright 2010-2017 Sarosys LLC <http://www.sarosys.com>
 
     This file is part of the Arachni Framework project and is subject to
     redistribution and commercial restrictions. Please see the Arachni Framework

data/bin/arachni_multi

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2016 Tasos Laskos <tasos.laskos@arachni-scanner.com>
+    Copyright 2010-2017 Sarosys LLC <http://www.sarosys.com>
 
     This file is part of the Arachni Framework project and is subject to
     redistribution and commercial restrictions. Please see the Arachni Framework
@@ -10,4 +10,9 @@
 require_relative '../lib/arachni'
 require_relative '../ui/cli/rpc/client/local'
 
+if Arachni.windows?
+    Arachni::UI::Output.print_error "This interface is not available on MS Windows."
+    exit
+end
+
 Arachni::UI::CLI::RPC::Client::Local.new

data/bin/arachni_reporter

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2016 Tasos Laskos <tasos.laskos@arachni-scanner.com>
+    Copyright 2010-2017 Sarosys LLC <http://www.sarosys.com>
 
     This file is part of the Arachni Framework project and is subject to
     redistribution and commercial restrictions. Please see the Arachni Framework

data/bin/arachni_reproduce

@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby
+=begin
+    Copyright 2010-2017 Sarosys LLC <http://www.sarosys.com>
+
+    This file is part of the Arachni Framework project and is subject to
+    redistribution and commercial restrictions. Please see the Arachni Framework
+    web site for more information on licensing and terms of use.
+=end
+
+require_relative '../ui/cli/reproduce'
+
+Arachni::UI::CLI::Reproduce.new

data/bin/arachni_rest_server

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2016 Tasos Laskos <tasos.laskos@arachni-scanner.com>
+    Copyright 2010-2017 Sarosys LLC <http://www.sarosys.com>
 
     This file is part of the Arachni Framework project and is subject to
     redistribution and commercial restrictions. Please see the Arachni Framework

data/bin/arachni_restore

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2016 Tasos Laskos <tasos.laskos@arachni-scanner.com>
+    Copyright 2010-2017 Sarosys LLC <http://www.sarosys.com>
 
     This file is part of the Arachni Framework project and is subject to
     redistribution and commercial restrictions. Please see the Arachni Framework

data/bin/arachni_rpc

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 =begin
-    Copyright 2010-2016 Tasos Laskos <tasos.laskos@arachni-scanner.com>
+    Copyright 2010-2017 Sarosys LLC <http://www.sarosys.com>
 
     This file is part of the Arachni Framework project and is subject to
     redistribution and commercial restrictions. Please see the Arachni Framework
@@ -10,4 +10,9 @@
 require_relative '../lib/arachni'
 require_relative '../ui/cli/rpc/client/remote'
 
+if Arachni.windows?
+    Arachni::UI::Output.print_error "This interface is not available on MS Windows."
+    exit
+end
+
 Arachni::UI::CLI::RPC::Client::Remote.new