arachni 0.4.7 → 1.0

checksums.yaml

@@ -1,15 +1,7 @@
 ---
-!binary "U0hBMQ==":
-  metadata.gz: !binary |-
-    ZDliMzg1ZDM5Zjc3YjU2Njg5NmY0MDVmY2I5ZmYwYzNmNmIwMTRhZA==
-  data.tar.gz: !binary |-
-    ZGExNDM4ZjUyMzhhOTY2MTEyODY4MThjMjQ0MDE0Njg2MTBkNTY1Yw==
+SHA1:
+  metadata.gz: 0bbd91e8fb1bcbcf5164c22e187397214e1894ac
+  data.tar.gz: 16adc7946a2fa8da04a50b2417cd8b731d291652
 SHA512:
-  metadata.gz: !binary |-
-    MzMyMzU2Yjg3ZjY3ZmVhZmU4ODViN2NjMTE2NDRhZjdiNTBhN2VhYTdlMWEx
-    MjE5ZmFkNjIzNjNmYWVkOWZmMzI1ZDBhOWFlYjY4NmFjNmVmNmI2NmZhYzE3
-    MTAwZDhjMDFjYmE5YWJhYmE5YzZjOTA5ZGJmM2Q3MzEzZTYzMGQ=
-  data.tar.gz: !binary |-
-    ZTA0ZWFhNDE2Mjk2ODcwNjk0MWEyNjMyZGI3NGRmMTQyOGMxYTJkMzUwZDFj
-    Y2NlMzlmODNjNzdkNWJiODU0ZTllZjRmODQ1MjQ2Y2Y0ZTAwYzIxOGI2YjRj
-    ZTVlNDU4MGViZjY5ODU1ZmZlOTg0MzQyOTg4NDNmOGIwNjI0ZDY=
+  metadata.gz: 913913c30128c176a2e55614ee95530343fd6f300cdf3008cbd6df69cf93b78c2b9c87ea748ae330524061a32a160563fb1aafdfc2b06ad4dcf853e5f9eb8b50
+  data.tar.gz: 59bd68596db84ed440707cde96554f13d6397a52ca0025f9019073ba250bf0e32296adbebd50ac099c02262bff40ac5c27d9d3bcffeb0d2afe74666577816b3d

data/AUTHORS.md

@@ -1,3 +1,3 @@
 # Authors
 
-Tasos "Zapotek" Laskos <tasos.laskos@gmail.com>
+Tasos "Zapotek" Laskos <tasos.laskos@arachni-scanner.com>

data/CHANGELOG.md

@@ -1,5 +1,211 @@
 # ChangeLog
 
+## 1.0 _(August 29, 2014)_
+
+- Executables:
+    - Added
+        - `arachni_restore` (`UI::CLI::RestoredFramework`)
+            - Restores snapshots of suspended scans.
+            - Prints snapshot metadata.
+        - `arachni_report` (`UI::CLI::Report`)
+            - Creates reports from `.afr` files.
+    - `arachni` (`UI::CLI::Framework`)
+        - `Ctrl+C` (`SIGINT`) now aborts the scan.
+        - Hitting `Enter` now toggles between the progress message and the
+            command screens.
+        - Updated to provide access to the new suspend-to-disk feature.
+        - Moved reporting functionality to `arachni_report`.
+- `Framework`
+    - `#audit_page` -- Updated to perform DOM/JS/AJAX analysis on the page and
+        feed DOM page snapshots and new paths back to the `Framework`.
+    - `#stats` renamed to `#statistics` with the return hash cleaned-up.
+    - `#opts` renamed to `#options`.
+- `Session`
+    - Updated to support login forms which depend on DOM/Javascript.
+- Added `State` -- Stores and provides access to the system's state.
+    - `Plugins` -- Stores plugin runtime states when suspending.
+    - `HTTP` -- Stores client headers and cookies.
+    - `Audit` -- Stores audit operations.
+    - `ElementFilter` -- Stores seen elements.
+    - `Framework` -- Stores the `Framework` state.
+        - `RPC` -- Stores the `RPC::Server::Framework` state.
+- Added `Data` -- Stores and provides access to the system's data.
+    - `Issues` -- Stores logged `Issue` objects.
+    - `Plugins` -- Stores plugin results.
+    - `Session` -- Stores login configuration.
+    - `Framework` -- Stores the `Framework` audit workload.
+        - `RPC` -- Stores the `RPC::Server::Framework` audit workload.
+- Added `Snapshot`
+    - Dumps and loads `State` and `Data` to and from disk to suspend and restore
+        active scans.
+- Removed the `Spider`.
+    - The Framework has grown to encompass a process providing the same
+        functionality as a result of `Browser` analysis.
+- `Element`
+    - Cleaned up initializers.
+        - Now passed a single Hash argument with configuration options.
+    - Added `GenericDOM`
+        - Provides an interface similar to traditional elements in order for
+            generic DOM elements to be logged and assigned as vectors to issues.
+    - Added `LinkTemplate`
+        - Basing its vector identification and manipulation to a user-provided
+            template to satisfy cases like ModRewrite and similar.
+        - Including `#dom` pointing to a `Auditable::DOM` object handling browser-based
+            link submissions/audits.
+    - `Form`
+        - Added `#dom` pointing to a `Auditable::DOM` object handling browser-based
+            form submissions/audits.
+    - `Link`
+        - Added `#dom` pointing to a `Auditable::DOM` object handling browser-based
+            link submissions/audits.
+    - `Cookie`
+        - Added `#dom` pointing to a `Auditable::DOM` object handling browser-based
+            cookie submissions/audits.
+    - `Capabilities::Auditable`
+        - Removed `#use_anonymous_auditor`
+        - `#auditable` => `#inputs`
+        - `#orig` => `#default_inputs`
+        - `#opts` => `#audit_options`
+        - `#audit` - Callback now get passed the HTTP response and element mutation
+            instead of response, audit options and mutation -- options can now be
+            accessed via the element's `#audit_options` attribute.
+        - Added `DOM` -- To handle DOM submission/auditing of elements.
+        - Split into the following `Capabilities`:
+            - `Analyzable`
+                - `Timeout`
+                    - General refactoring and code cleanup.
+                    - Updated the algorithm to ensure server responsiveness before each phase.
+                    - Lowered the amount of performed requests.
+                    - No longer downloads response bodies.
+                - `RDiff` => `Differential`
+                - `Taint`
+            - `Submittable`
+            - `Inputtable`
+- `RPC`
+    - `Serializer` -- Replaced `Marshal` and `YAML` as RPC serialization providers.
+        - Delegates to `MessagePack`.
+        - Supports message compression -- applied based on message size to minimize overhead.
+    - `opts` handler renamed to `options`.
+    - `Server`
+        - `Dispatcher`
+            - `#dispatch` -- Returns `false` when the pool is empty as a signal
+                to check back later.
+            - Removed `#proc_info` method.
+            - Removed `proc` from job info data.
+            - `Handler` renamed to `Service`.
+        - `Instance`
+            - Removed `#output`.
+        - `Framework`
+            - Removed `#output`.
+            - `#progress`
+                - `:messages` now returns `Framework#status_messages` instead of
+                    output messages.
+                - Cleaned up return data.
+                - Removed `#progress_data` alias.
+- `HTTP` expanded to be a complete wrapper around Typhoeus, providing:
+    - `Headers`
+    - `Message`
+    - `Request`
+    - `Response`
+    - `Client`
+        - `#request` options:
+            - `:params` => `:parameters`
+            - `:async` => `:mode` (with values of `:async` and `:sync`)
+            - Added `:http_max_response_size`.
+    - `ProxyServer` -- Moved the proxy server out of the `Proxy` plugin and
+        updated it to work with `Arachni::HTTP` objects.
+- `Browser` -- Real browser driver providing DOM/JS/AJAX support.
+- `BrowserCluster` -- Maintains a pool of `Arachni::Browser` instances
+    and distributes the analysis workload of multiple resources.
+- `Page`
+    - Cleaned-up attributes.
+    - Attributes (`#links`, `#forms`, `#paths` etc.) are lazy-parsed on-demand.
+    - Added:
+        - `#response` -- Associated `HTTP::Response`.
+        - `#dom` -- Associated `Arachni::Page::DOM`.
+- `Page::DOM` -- Static DOM snapshot as computed by a real browser.
+- `Parser` -- Updated to **only** operate under the context of the
+    `HTTP::Response` with which it was initialized -- no longer supports parsing
+    data from external sources.
+- `Options` -- Rewritten with renamed option names and grouped relevant options together.
+- `Report` (Renamed from `AuditStore`)
+    - `#save` -- Updated to store a compressed `Marshal` dump of the instance.
+    - `.load` -- Updated to load the new `#save` format.
+- `Component::Options` -- Refactored initializers and API.
+    - `Enum` renamed to `MultipleChoice`.
+- `Reporters` (Renamed from `Reports`)
+    - Removed `metareport`.
+    - All updated to the new format.
+- Plugins
+    - Descriptions have been converted to GitHub-flavored Markdown.
+    - `resolver` -- Removed as the report now contains that information in the
+        responses associated with each issue.
+    - `proxy`
+        - Updated to use `HTTP::ProxyServer`.
+        - Added `ignore_responses` option.
+            - Forces the proxy to only extract vector information from observed
+                HTTP requests and not analyze responses.
+    - `autologin`
+        - `params` option renames to `parameters`.
+        - Changed results to include `status` (`String`) and `message` (`String`)
+            instead of `code` (`Integer`) and `msg` (`String`).
+        - Updated to abort the scan upon login failure.
+    - `content_types`
+        - Renamed `params` in logged results to `parameters`.
+    - `cookie_collector`
+        - Renamed `res` in logged results to `response`.
+    - `waf_detector`
+        - Changed results to include `status` (`Symbol`) and `message` (`String`)
+            instead of `code` (`Integer`) and `msg` (`String`).
+    - `healthmap`
+        - Changed results to use `with_issues` and `without_issues` instead of
+            `unsafe` and `safe`.
+- Path extractors
+    - Added:
+        - Extract partial paths from HTML comments (`comments`).
+        - `script` - Extract partial paths from scripts.
+- Moved all Framework components (`modules`, `plugins`, `reports`, etc.)
+    under `components/`.
+- Renamed `modules` to `checks`, also:
+    - _Audit_ checks renamed to _Active_ checks.
+    - _Recon_ checks renamed to _Passive_ checks.
+- Checks
+    - Descriptions and `remedy_guidance` have been converted to GitHub-flavored Markdown.
+    - Renamed
+        - `xpath` => `xpath_injection`
+        - `ldapi` => `ldap_injection`
+        - `sqli` => `sql_injection`
+        - `sqli_blind_rdiff` => `sql_injection_differential`
+        - `sqli_blind_timing` => `sql_injection_timing`
+        - `htaccess` => `htaccess_limit`
+    - Active
+        - New
+            - `xss_dom` -- Injects HTML code via DOM-based links, forms and cookies.
+            - `xss_dom_inputs` -- Injects HTML code via orphan text inputs with
+                associated DOM events.
+            - `xss_dom_script_context` -- Injects JavaScript code via DOM-based
+                links, forms and cookies.
+            - `no_sql_injection` -- NoSQL Injection (error-based) .
+            - `no_sql_injection_differential` -- Blind NoSQL Injection (differential analysis).
+        - `xss` -- Added support for Browser-based taint-analysis.
+        - `xss_script_context` -- Added support for Browser-based taint-analysis.
+            - Renamed from `xss_script_tag`.
+        - `unvalidated_redirect` -- Updated to also use full browser evaluation
+            in order to detect JS redirects.
+        - `os_cmd_injection` -- Added payloads for *BSD and AIX.
+    - Passive
+        - New
+            - `backup_directories` -- Backup directories.
+            - `cookie_set_for_parent_domain` -- Cookie set for parent domain.
+            - Grep
+                - `hsts` - Checks HTTPS pages for missing `Strict-Transport-Security` headers.
+        - `backup_files` -- Updated filename formats.
+        - `x_forwarded_for_access_restriction_bypass` renamed to `origin_spoof_access_restriction_bypass`.
+            - Also updated to use more origin headers.
+        - Grep
+            - `emails` - Updated to handle simple (`[at]` and `[dot]`) obfuscation.
+            - `insecure_cookies` - Only check HTTPS pages.
+
 ## 0.4.7 _(April 12, 2014)_
 
 - `Spider`
@@ -104,6 +310,10 @@
             - Check for an ASP platform instead of a Windows one.
             - Fixed `LocalJumpError`.
 - Plugins
+    - Removed
+        - `libnotify`
+        - `profiler`
+        - `rescan`
     - `autologin`
         - Changed `print_bad` to `print_error` so that errors are written to the
             error log.
@@ -442,6 +652,7 @@
   - Protocol -- Now supports both ```Marshal``` and ```YAML``` automatically.
       - ```Marshal``` by default since it's many times faster than ```YAML```.
       - ```YAML``` as an automatic fallback in order to maintain backwards compatibility and ease of integration with 3rd parties.
+          - Updated to use the Ruby-default ```Psych``` engine.
   - ```Framework```
       - Updated gathering of slave status -- once a slave is done it reports back to the master.
       - Clean-up happens automatically, clients no longer need to call ```#clean_up``` (like previously mentioned).

data/CONTRIBUTORS.md

@@ -15,6 +15,7 @@ suggestions or testing it.
 - [Evan Beard](mailto:beard.evan@gmail.com) for feedback and patches.
 - [Michael Borohovski](mailto:borski@mit.edu) for testing, feedback and patches.
 - [Ben Sedat](mailto:bsedat@alum.mit.edu) for testing, feedback and patches.
+- [Simon Treadaway](mailto:ssgtreadaway@outlook.com) for testing and feedback.
 - [Michiel van Es](mailto:mve@pragmasec.nl) for relentless testing and feedback.
 
 A big thanks to my buddy [Andreas](mailto:rainmakergr@gmail.com) for the original

data/Gemfile

@@ -1,6 +1,27 @@
 source 'https://rubygems.org'
 
-gem 'yard'
-gem 'redcarpet'
+gem 'rake'
+
+group :docs do
+    gem 'yard'
+    gem 'redcarpet'
+end
+
+group :spec do
+    gem 'simplecov', require: false, group: :test
+
+    gem 'rspec', '2.99'
+    gem 'faker'
+
+    gem 'puma' if !Gem.win_platform? || RUBY_PLATFORM == 'java'
+
+    gem 'sinatra'
+    gem 'sinatra-contrib'
+end
+
+group :prof do
+    gem 'stackprof'
+end
 
 gemspec
+

data/LICENSE.md

@@ -1,5 +1,38 @@
 # License
 
+Copyright 2010-2014 [Tasos Laskos](mailto:tasos.laskos@arachni-scanner.com).
+
+The Arachni Framework (henceforth referred to simply as "Arachni") is dual-licensed.
+
+Cases that include **commercialization** of Arachni require a commercial,
+non-free license. Otherwise, the system can be used under the terms of
+Apache License v2.0, found at the bottom of this document.
+
+Cases of **commercialization** are:
+
+* Using Arachni to provide commercial managed/Software-as-a-Service services.
+* Distributing Arachni as a commercial product or as part of one.
+
+Cases which **don't** require a commercial license, and thus fall under
+the terms of Apache License v2.0, include (but are not limited to):
+
+* Penetration testers (or penetration testing organizations) using Arachni as
+    part of their assessment toolkit.
+    * So long as that doesn't conflict with the **commercialization** clause.
+* Using Arachni to test your own systems.
+* Any non-commercial use of Arachni.
+
+If you need to acquire a commercial license or are unsure about whether you
+need to acquire a commercial license, please get in touch, we'll be happy to
+clarify things for you and work with you to accommodate your requirements.
+
+You can use the [licensing contact form](http://www.arachni-scanner.com/license)
+to contact us about these matters.
+
+Code contributions will be accepted under the Apache License v2.0.
+
+## Apache License v2.0
+
 ```
                                  Apache License
                            Version 2.0, January 2004
@@ -175,4 +208,4 @@
       defend, and hold each Contributor harmless for any liability
       incurred by, or claims asserted against, such Contributor by reason
       of your accepting any such warranty or additional liability.
-```
+```

data/README.md

@@ -1,9 +1,17 @@
+**NOTICE**:
+
+* Arachni's license has changed, please see the _LICENSE_ file before working
+    with the project.
+* v1.0 is not backwards compatible.
+
+<hr/>
+
 # Arachni - Web Application Security Scanner Framework
 
 <table>
     <tr>
         <th>Version</th>
-        <td>0.4.7</td>
+        <td>1.0</td>
     </tr>
     <tr>
         <th>Homepage</th>
@@ -30,7 +38,7 @@
     </tr>
     <tr>
        <th>Author</th>
-       <td><a href="mailto:tasos.laskos@gmail.com">Tasos Laskos</a> (<a href="http://twitter.com/Zap0tek">@Zap0tek</a>)</td>
+       <td><a href="mailto:tasos.laskos@arachni-scanner.com">Tasos Laskos</a> (<a href="http://twitter.com/Zap0tek">@Zap0tek</a>)</td>
     </tr>
     <tr>
         <th>Twitter</th>
@@ -42,7 +50,7 @@
     </tr>
     <tr>
         <th>License</th>
-        <td><a href="file.LICENSE.html">Apache License Version 2.0</a></td>
+        <td>Dual-licensed (Apache License v2.0/Commercial) - (see LICENSE file)</td>
     </tr>
 </table>
 
@@ -54,20 +62,19 @@ Arachni is an Open Source, feature-full, modular, high-performance Ruby framewor
 aimed towards helping penetration testers and administrators evaluate the security
 of web applications.
 
-It is smart, it trains itself by learning from the HTTP responses it receives
-during the audit process and is able to perform meta-analysis using a number of
+It is smart, it trains itself by monitoring and learning from the web application's
+behavior during the scan process and is able to perform meta-analysis using a number of
 factors in order to correctly assess the trustworthiness of results and intelligently
-identify false-positives.
+identify (or avoid) false-positives.
 
 Unlike other scanners, it takes into account the dynamic nature of web applications,
 can detect changes caused while travelling through the paths of a web application’s
-cyclomatic complexity and is able to adjust itself accordingly. This way attack/input
-vectors that would otherwise be undetectable by non-humans are seamlessly handled by Arachni.
+cyclomatic complexity and is able to adjust itself accordingly. This way, attack/input
+vectors that would otherwise be undetectable by non-humans can be handled seamlessly.
 
-Moreover, Arachni yields great performance due to its asynchronous HTTP model
-(courtesy of Typhoeus) — especially when combined with a High Performance Grid
-setup which allows you to combine the resources of multiple nodes for lightning
-fast scans. Thus, you’ll only be limited by the responsiveness of the server under audit.
+Moreover, due to its integrated browser environment, it can also audit and inspect
+client-side code, as well as support highly complicated web applications which make
+heavy use of technologies such as JavaScript, HTML5, DOM manipulation and AJAX.
 
 Finally, it is versatile enough to cover a great deal of use cases, ranging from
 a simple command line scanner utility, to a global high performance grid of
@@ -75,14 +82,14 @@ scanners, to a Ruby library allowing for scripted audits, to a multi-user
 multi-scan web collaboration platform.
 
 **Note**: Despite the fact that Arachni is mostly targeted towards web application
-security, it can easily be used for  general purpose scraping, data-mining, etc
-with the addition of custom modules.
+security, it can easily be used for general purpose scraping, data-mining, etc.
+with the addition of custom components.
 
 ### Arachni offers:
 
 #### A stable, efficient, high-performance framework
 
-Module, report and plugin writers are allowed to easily and quickly create and
+`Check`, `report` and `plugin` developers are allowed to easily and quickly create and
 deploy their components with the minimum amount of restrictions imposed upon them,
 while provided with the necessary infrastructure to accomplish their goals.
 
@@ -120,24 +127,118 @@ you with its findings.
  - User Agent spoofing.
  - Proxy support for SOCKS4, SOCKS4A, SOCKS5, HTTP/1.1 and HTTP/1.0.
  - Proxy authentication.
- - Site authentication (Automated form-based, Cookie-Jar, Basic-Digest, NTLM and others).
- - Automatic log-out detection and re-login during the audit (when the initial
-    login was performed via the AutoLogin plugin).
+ - Site authentication (Automated form-based, Cookie-Jar, Basic-Digest, NTLMv1 and others).
+ - Automatic log-out detection and re-login during the scan (when the initial
+    login was performed via the `autologin` or `proxy` plugins).
  - Custom 404 page detection.
  - UI abstraction:
     - [Command-line Interface](https://github.com/Arachni/arachni/wiki/Executables).
     - [Web User Interface](https://github.com/Arachni/arachni-ui-web).
  - Pause/resume functionality.
+ - Hibernation support -- Suspend to and restore from disk.
  - High performance asynchronous HTTP requests.
     - With adjustable concurrency.
+    - With the ability to auto-detect server health and adjust its concurrency
+        automatically.
+ - Support for custom default input values, using pairs of patterns (to be matched
+    against input names) and values to be used to fill in matching inputs.
+
+### Integrated browser environment
+
+Arachni includes an integrated, real browser environment in order to provide
+sufficient coverage to modern web applications which make use of technologies
+such as HTML5, JavaScript, DOM manipulation, AJAX, etc.
+
+In addition to the monitoring of the vanilla DOM and JavaScript environments,
+Arachni's browsers also hook into popular frameworks to make the logged data
+easier to digest:
+
+- [JQuery](http://jquery.com/)
+- [AngularJS](https://angularjs.org/)
+- More to come...
+
+In essence, this turns Arachni into a DOM and JavaScript debugger, allowing it to
+monitor DOM events and JavaScript data and execution flows. As a result, not only
+can the system trigger and identify DOM-based issues, but it will accompany them
+with a great deal of information regarding the state of the page at the time.
+
+Relevant information include:
+
+ - Page DOM, as HTML code.
+     - With a list of DOM transitions required to restore the state of the
+         page to the one at the time it was logged.
+ - Original DOM (i.e. prior to the action that caused the page to be logged),
+     as HTML code.
+     - With a list of DOM transitions.
+ - Data-flow sinks -- Each sink is a JS method which received a tainted argument.
+     - Parent object of the method (ex.: `DOMWindow`).
+     - Method signature (ex.: `decodeURIComponent()`).
+     - Arguments list.
+         - With the identified taint located recursively in the included objects.
+     - Method source code.
+     - JS stacktrace.
+ - Execution flow sinks -- Each sink is a successfully executed JS payload,
+     as injected by the security checks.
+     - Includes a JS stacktrace.
+ - JavaScript stack-traces include:
+     - Method names.
+     - Method locations.
+     - Method source codes.
+     - Argument lists.
+
+In essence, you have access to roughly the same information that your favorite
+debugger (for example, FireBug) would provide, as if you had set a breakpoint to
+take place at the right time for identifying an issue.
+
+#### Browser-cluster
+
+The browser-cluster is what coordinates the browser analysis of resources and
+allows the system to perform operations which would normally be quite time
+consuming in a high-performance fashion.
+
+Configuration options include:
+
+ - Adjustable pool-size, i.e. the amount of browser workers to utilize.
+ - Timeout for each job.
+ - Worker TTL counted in jobs -- Workers which exceed the TTL have their browser
+     process respawned.
+ - Ability to disable loading images.
+ - Adjustable screen width and height.
+     - Can be used to analyze responsive and mobile applications.
+
+### Coverage
+
+The system can provide great coverage to modern web applications due to its
+integrated browser environment. This allows it to interact with complex applications
+that make heavy use of client-side code (like JavaScript) just like a human would.
+
+In addition to that, it also knows about which browser state changes the application
+has been programmed to handle and is able to trigger them programatically in
+order to provide coverage for a full set of possible scenarios.
+
+By inspecting all possible pages and their states (when using client-side code)
+Arachni is able to extract and audit the following elements and their inputs:
+
+ - Forms
+    - Along with ones that require interaction with a real browser due to DOM events.
+ - Links
+    - Along with ones that have client-side parameters in their fragment, i.e.:
+        `http://example.com/#/?param=val&param2=val2`
+    - With support for rewrite rules.
+ - LinkTemplates -- Allowing for extraction of arbitrary inputs from generic paths,
+    based on user-supplied templates -- useful when rewrite rules are not available.
+    - Along with ones that have client-side parameters in their URL fragments, i.e.:
+            `http://example.com/#/param/val/param2/val2`
+ - Cookies
+ - Headers
+ - Generic client-side elements like `input`s which have associated DOM events.
+ - AJAX-request parameters.
 
 ### Open [distributed architecture](https://github.com/Arachni/arachni/wiki/Distributed-components)
 
-- High-performance/low-bandwidth [communication protocol](https://github.com/Arachni/arachni-rpc-em).
-    - `Marshal` serialization for performance and efficiency.
-        - Automatically falls back to `YAML` for ease of integration with 3rd party systems.
-    - TCP/IP for general network communications.
-    - UNIX domain sockets for multi-Instance IPC.
+- High-performance/low-bandwidth [communication protocol](https://github.com/Arachni/arachni-rpc).
+    - `MessagePack` serialization for performance, efficiency and ease of
+        integration with 3rd party systems.
 - Remote monitoring and management of Dispatchers and Instances.
 - Parallel scans -- Each scan is compartmentalized to its own OS process to take
     advantage of:
@@ -148,51 +249,71 @@ you with its findings.
     Instances to:
     - Take advantage of multi-core/SMP architectures.
     - Greatly diminish scan-times.
-- Dispatcher Grids supporting:
-    - _(Optional)_ High-Performance mode -- Combines the resources of
-        multiple nodes to perform multi-Instance scans.
-        - Enabled on a per-scan basis.
+- Dispatcher Grid:
+    - Self-healing.
+    - Scale up/down by hot-plugging/hot-unplugging nodes.
+        - Can scale up infinitely by adding nodes to increase scan capacity.
     - _(Always-on)_ Load-balancing -- All Instances are automatically provided
         by the least burdened Grid member.
         - With optional per-scan opt-out/override.
+    - _(Optional)_ High-Performance mode -- Combines the resources of
+        multiple nodes to perform multi-Instance scans.
+        - Enabled on a per-scan basis.
 - SSL encryption (with optional peer authentication).
 
-### Crawler
+### Scope configuration
 
- - Filters for redundant pages like galleries, catalogs, etc based on regular
+ - Filters for redundant pages like galleries, catalogs, etc. based on regular
     expressions and counters.
     - Can optionally detect and ignore redundant pages automatically.
- - URL exclusion filter using regular expressions.
- - Page exclusion filter based on content, using regular expressions.
- - URL inclusion filter using regular expressions.
+ - URL exclusion filters using regular expressions.
+ - Page exclusion filters based on content, using regular expressions.
+ - URL inclusion filters using regular expressions.
  - Can be forced to only follow HTTPS paths and not downgrade to HTTP.
  - Can optionally follow subdomains.
- - Adjustable link count limit.
+ - Adjustable page count limit.
  - Adjustable redirect limit.
- - Adjustable depth limit.
- - Modular path extraction via "Path Extractor" components.
+ - Adjustable directory depth limit.
+ - Adjustable DOM depth limit.
+ - Adjustment using URL-rewrite rules.
  - Can read paths from multiple user supplied files (to both restrict and extend
-    the scope of the crawl).
+    the scope).
 
-### Auditor
+### Audit
 
  - Can audit:
     - Forms
-        - Can refresh nonce tokens.
+        - Can automatically refresh nonce tokens.
+        - Can submit them via the integrated browser environment.
     - Links
+        - Can load them via the integrated browser environment.
+    - LinkTemplates
+        - Can load them via the integrated browser environment.
     - Cookies
+        - Can load them via the integrated browser environment.
     - Headers
+    - Generic client-side DOM elements like `input`s.
  - Can ignore binary/non-text pages.
- - Can optionally audit forms and links using both `GET` and `POST` HTTP methods.
+ - Can optionally audit elements using both `GET` and `POST` HTTP methods.
  - Can optionally submit all links and forms of the page along with the cookie
     permutations to provide extensive cookie-audit coverage.
  - Can exclude specific input vectors by name.
+ - Can include specific input vectors by name.
 
-### Platform fingerprinter
+### Components
 
-In order to make efficient use of the available bandwidth, Arachni performs some
-basic platform fingerprinting and tailors the audit process to the server-side
-deployed platforms by only injecting applicable payloads.
+Arachni is a highly modular system, employing several components of distinct
+types to perform its duties.
+
+In addition to enabling or disabling the bundled components so as to adjust the
+system's behavior and features as needed, functionality can be extended via the
+addition of user-created components to suit almost every need.
+
+#### Platform fingerprinters
+
+In order to make efficient use of the available bandwidth, Arachni performs
+rudimentary platform fingerprinting and tailors the audit process to the server-side
+deployed technologies by only using applicable payloads.
 
 Currently, the following platforms can be identified:
 
@@ -225,34 +346,15 @@ can be disabled altogether.
 Finally, Arachni will always err on the side of caution and send all available
 payloads when it fails to identify specific platforms.
 
-### HTML Parser
-
-Can extract and analyze:
+#### Checks
 
- - Forms
- - Links
- - Cookies
- - Headers
-
-###  Module Management
+_Checks_ are system components which perform security checks and log issues.
 
- - Very simple and easy to use module API providing access to multiple levels of complexity.
- - Helper audit methods:
-    - For form, link, cookie and header auditing.
-    - A wide range of injection strings/input combinations.
-    - For taint analysis, timing attacks, differential analysis, server-side
-        file/directory detection and more.
-    - Writing RFI, SQL injection, XSS etc modules is a matter of minutes, if not seconds.
+##### Active
 
-#### Available modules (security checks)
+Active checks engage the web application via its inputs.
 
-Modules are system components which perform security checks and log issues.
-
-##### Audit (Active)
-
-Audit modules actively engage the web application via its inputs.
-
-- SQL injection (`sqli`) -- Error based vulnerability detection.
+- SQL injection (`sql_injection`) -- Error based detection.
     - Oracle
     - ColdFusion
     - InterBase
@@ -270,11 +372,14 @@ Audit modules actively engage the web application via its inputs.
     - Ingres
     - HSQLDB
     - MS Access
-- Blind SQL injection using rDiff analysis (`sqli_blind_rdiff`).
-- Blind SQL injection using timing attacks (`sqli_blind_timing`).
+- Blind SQL injection using differential analysis (`sql_injection_differential`).
+- Blind SQL injection using timing attacks (`sql_injection_timing`).
     - MySQL
     - PostgreSQL
     - MSSQL
+- NoSQL injection (`no_sql_injection`) -- Error based vulnerability detection.
+    - MongoDB
+- Blind NoSQL injection using differential analysis (`no_sql_injection_differential`).
 - CSRF detection (`csrf`).
 - Code injection (`code_injection`).
     - PHP
@@ -288,7 +393,7 @@ Audit modules actively engage the web application via its inputs.
     - Python
     - JSP
     - ASP.NET
-- LDAP injection (`ldapi`).
+- LDAP injection (`ldap_injection`).
 - Path traversal (`path_traversal`).
     - *nix
     - Windows
@@ -302,6 +407,8 @@ Audit modules actively engage the web application via its inputs.
 - Response splitting (`response_splitting`).
 - OS command injection (`os_cmd_injection`).
     - *nix
+    - *BSD
+    - IBM AIX
     - Windows
 - Blind OS command injection using timing attacks (`os_cmd_injection_timing`).
     - Linux
@@ -310,7 +417,7 @@ Audit modules actively engage the web application via its inputs.
     - Windows
 - Remote file inclusion (`rfi`).
 - Unvalidated redirects (`unvalidated_redirect`).
-- XPath injection (`xpath`).
+- XPath injection (`xpath_injection`).
     - Generic
     - PHP
     - Java
@@ -320,15 +427,16 @@ Audit modules actively engage the web application via its inputs.
 - Path XSS (`xss_path`).
 - XSS in event attributes of HTML elements (`xss_event`).
 - XSS in HTML tags (`xss_tag`).
-- XSS in HTML 'script' tags  (`xss_script_tag`).
+- XSS in "script" context (`xss_script_context`).
 - Source code disclosure (`source_code_disclosure`)
 
-##### Recon (Passive)
+##### Passive
 
-Recon modules look for the existence of files, folders and signatures.
+Passive checks look for the existence of files, folders and signatures.
 
 - Allowed HTTP methods (`allowed_methods`).
 - Back-up files (`backup_files`).
+- Backup directories (`backup_directories`)
 - Common directories (`common_directories`).
 - Common files (`common_files`).
 - HTTP PUT (`http_put`).
@@ -349,52 +457,39 @@ Recon modules look for the existence of files, folders and signatures.
 - Insecure cookies (`insecure_cookies`).
 - HttpOnly cookies (`http_only_cookies`).
 - Auto-complete for password form fields (`password_autocomplete`).
-- X-Forwarded-For Access Restriction Bypass (`x_forwarded_for_access_restriction_bypass`)
+- Origin Spoof Access Restriction Bypass (`origin_spoof_access_restriction_bypass`)
 - Form-based upload (`form_upload`)
 - localstart.asp (`localstart_asp`)
+- Cookie set for parent domain (`cookie_set_for_parent_domain`)
+- Missing `Strict-Transport-Security` headers for HTTPS sites (`hsts`).
 
-### Report Management
-
- - Modular design.
-
-#### Available reports
+#### Reporters
 
 - Standard output
-- HTML (`html`).
-- XML (`xml`).
-- TXT (`text`).
-- AFR (`afr`) -- The default Arachni Framework Report format.
-- JSON (`json`)
-- Marshal (`marshal`)
-- YAML (`yaml`)
-- Metareport  (`metareport`) -- Providing Metasploit integration to allow for
-    [automated and assisted exploitation](http://arachni.github.com/arachni/file.EXPLOITATION.html).
-
-### Plug-in Management
-
- - Modular design.
- - Plug-ins are framework demi-gods, they have direct access to the framework instance.
- - Can be used to add abstract functionality to Arachni.
-
-#### Available plugins
+- [HTML](http://downloads.arachni-scanner.com/dev/reports/report.html/)
+    ([zip](http://downloads.arachni-scanner.com/dev/reports/report.html.zip)) (`html`).
+- [XML](http://downloads.arachni-scanner.com/dev/reports/report.xml) (`xml`).
+- [Text](http://downloads.arachni-scanner.com/dev/reports/report.txt) (`text`).
+- [JSON](http://downloads.arachni-scanner.com/dev/reports/report.json) (`json`)
+- [Marshal](http://downloads.arachni-scanner.com/dev/reports/report.marshal) (`marshal`)
+- [YAML](http://downloads.arachni-scanner.com/dev/reports/report.yml) (`yaml`)
+- [AFR](http://downloads.arachni-scanner.com/dev/reports/report.afr) (`afr`)
+    - The default Arachni Framework Report format.
+
+#### Plugins
 
 Plugins add extra functionality to the system in a modular fashion, this way the
 core remains lean and makes it easy for anyone to add arbitrary functionality.
 
-- ReScan  (`rescan`)-- It uses the AFR report of a previous scan to extract the sitemap
-    in order to avoid a redundant crawl.
 - Passive Proxy  (`proxy`) -- Analyzes requests and responses between the web app and
     the browser assisting in AJAX audits, logging-in and/or restricting the scope of the audit.
 - Form based AutoLogin (`autologin`).
 - Dictionary attacker for HTTP Auth (`http_dicattack`).
 - Dictionary attacker for form based authentication (`form_dicattack`).
-- Profiler (`profiler`) -- Performs taint analysis (with benign inputs) and response time analysis.
 - Cookie collector (`cookie_collector`) -- Keeps track of cookies while establishing a timeline of changes.
 - WAF (Web Application Firewall) Detector (`waf_detector`) -- Establishes a baseline of
     normal behavior and uses rDiff analysis to determine if malicious inputs cause any behavioral changes.
 - BeepNotify (`beep_notify`) -- Beeps when the scan finishes.
-- LibNotify (`libnotify`) -- Uses the libnotify library to send notifications for each
-    discovered issue and a summary at the end of the scan.
 - EmailNotify (`email_notify`) -- Sends a notification (and optionally a report) over SMTP at
     the end of the scan.
 - VectorFeed (`vector_feed`) -- Reads in vector data from which it creates elements to be
@@ -406,7 +501,7 @@ core remains lean and makes it easy for anyone to add arbitrary functionality.
 - Content-types (`content_types`) -- Logs content-types of server responses aiding in the
     identification of interesting (possibly leaked) files.
 
-#### Defaults
+##### Defaults
 
 Default plugins will run for every scan and are placed under `/plugins/defaults/`.
 
@@ -415,7 +510,7 @@ Default plugins will run for every scan and are placed under `/plugins/defaults/
 - Healthmap (`healthmap`) -- Generates sitemap showing the health of each crawled/audited URL
 - Resolver (`resolver`) -- Resolves vulnerable hostnames to IP addresses.
 
-##### Meta
+###### Meta
 
 Plugins under `/plugins/defaults/meta/` perform analysis on the scan results
 to determine trustworthiness or just add context information or general insights.
@@ -424,7 +519,7 @@ to determine trustworthiness or just add context information or general insights
     when the affected audited pages returned unusually high response times to begin with.
     It also points out the danger of DoS attacks against pages that perform heavy-duty processing.
 - Discovery (`discovery`) -- Performs anomaly detection on issues logged by discovery
-    modules and warns of the possibility of false positives where applicable.
+    checks and warns of the possibility of false positives where applicable.
 - Uniformity (`uniformity`) -- Reports inputs that are uniformly vulnerable across a number
     of pages hinting to the lack of a central point of input sanitization.
 
@@ -433,36 +528,25 @@ to determine trustworthiness or just add context information or general insights
 The Trainer is what enables Arachni to learn from the scan it performs and
 incorporate that knowledge, on the fly, for the duration of the audit.
 
-Modules have the ability to individually force the Framework to learn from the
+Checks have the ability to individually force the Framework to learn from the
 HTTP responses they are going to induce.
 
 However, this is usually not required since Arachni is aware of which requests
-are more likely to uncover new elements or attack vectors and will adapt itself accordingly.
+are more likely to uncover new elements or attack vectors and will adapt itself
+accordingly.
 
-Still, this can be an invaluable asset to Fuzzer modules.
+Still, this can be an invaluable asset to Fuzzer checks.
 
 ## [Installation](https://github.com/Arachni/arachni/wiki/Installation)
 
 ## [Usage](https://github.com/Arachni/arachni/wiki/User-guide)
 
-## Configuration of _extras_
-
-The _extras_ directory holds components that are considered too specialised,
-dangerous or in some way unsuitable for utilising without explicit user interaction.
-
-This directory was mainly added to distribute modules which can be helpful but
-should not be put in the default _modules_ directory to prevent them from
-being automatically loaded.
-
-Should you want to use these extra components simply move them from the
-_extras_ folder to their appropriate system directories.
-
 ## Running the specs
 
 You can run `rake spec` to run **all** specs or you can run them selectively using the following:
 
     rake spec:core            # for the core libraries
-    rake spec:modules         # for the modules
+    rake spec:checks          # for the checks
     rake spec:plugins         # for the plugins
     rake spec:reports         # for the reports
     rake spec:path_extractors # for the path extractors
@@ -470,7 +554,7 @@ You can run `rake spec` to run **all** specs or you can run them selectively usi
 **Please be warned**, the core specs will require a beast of a machine due to the
 necessity to test the Grid/multi-Instance features of the system.
 
-**Note**: _The module specs will take about 90 minutes due to the timing-attack tests._
+**Note**: _The check specs will take about 90 minutes due to the timing-attack tests._
 
 ## Bug reports/Feature requests
 
@@ -486,7 +570,7 @@ We're happy to accept help from fellow code-monkeys and these are the steps you
 need to follow in order to contribute code:
 
 * Fork the project.
-* Start a feature branch based on the [experimental](https://github.com/Arachni/arachni-ui-web/tree/experimental)
+* Start a feature branch based on the [experimental](https://github.com/Arachni/arachni/tree/experimental)
     branch (`git checkout -b <feature-name> experimental`).
 * Add specs for your code.
 * Run the spec suite to make sure you didn't break anything (`rake spec:core`
@@ -496,11 +580,11 @@ need to follow in order to contribute code:
 
 ## License
 
-Arachni is licensed under the Apache License Version 2.0.<br/>
-See the [LICENSE](file.LICENSE.html) file for more information.
+Dual-licensed (Apache License v2.0/Commercial) -- please see the _LICENSE_ file
+for more information.
 
 ## Disclaimer
 
 This is free software and you are allowed to use it as you see fit.
-However, neither the development team nor any of our contributors can held
-responsible for your actions or for any damage caused by the use of this software.
+However, neither the development team nor any of our contributors can be held
+responsible for your actions nor for any damage caused by the use of this software.