arachni 1.5 → 1.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 9258760c9e75d398da9ab655640904e5d51544c4
-  data.tar.gz: b7f5f56e8f1977916b7a87465954c354892cc4d3
+SHA256:
+  metadata.gz: bf786db138bf57e440712a7fab2a8b4cd4fb97f1e908a5ec29a41889daec526f
+  data.tar.gz: 7485ca9d6093da52bd4d5d8f6c139a6fa4305719cbd518af4d37455a84ebd723
 SHA512:
-  metadata.gz: baef5531633d799813d4bbfdfc89a44aa682feb5b502349e40c19ec9036ef074f2cf4f8838403487e79b2ac905ace671863b5ee092515251418d0180f0db659b
-  data.tar.gz: 43e379175dff426bc078a5055dda4b0d65dee05f47df7032559501ce1603d5e4c32517ff0c9fb69669a0d21b7710df4f62b2899de099e32ebcc530165a039c46
+  metadata.gz: f7ce0f442dd3dfc68dba6eb331703ec75cc5d57021b5cb2daee5acd6050fa8f9249bfcdb4b697fc5fe9f48bdf2c34530c4b4af4bed5e69455b78f070d36520ff
+  data.tar.gz: 73392cf4dd5ea8e2fbd183beb6b7ef41331313e79fdfdc2c5c06012f6ec935c1bb82a065c16b9437e268a0cecf827f92b89b5a50914d2ef93adae931403b1d7c

data/CHANGELOG.md

@@ -1,5 +1,68 @@
 # ChangeLog
 
+## 1.6.1 _(March 20, 2022)_
+
+ - `Browser#eelenium` -- Disable sandbox.
+
+## 1.6.0 _(March 4, 2022)_
+
+- Options
+    - Scope
+        - `--scope-dom-depth-limit` -- Changed default from `5` to `4`.
+        - `--scope-auto-redundant` -- Changed default from `inf` to `15`.
+        - `--scope-directory-depth-limit` -- Changed default from `inf` to `10`.
+    - Browser cluster
+        - `--browser-cluster-pool-size` -- Changed default from `5` to `4`.
+    - HTTP
+       - `--http-max-concurrency` -- Changed default from `20` to `10`.
+    - New
+        - `--scope-dom-event-inheritance-limit` -- Limits the amount of inherited events.
+        - `--browser-cluster-session-storage` -- Sets the browsers' session storage.
+        - `--browser-cluster-wait-for-timers` -- Wait for the maximum `setTimeout()`
+           -- Used to be hardcoded to `true`, now defaults to `false`.
+- `URI`
+    - `#encode` -- Fixed encoding order of `+`.
+    - `#decode` -- Fixed decoding order of `+`.
+- `Element`
+    - Added `NestedCookie`: Handles key-value pairs inside individual cookies.
+- `Browser` -- Replaced PhantomJS with headless Chrome.
+    - `Javascript`
+        - `DOMMonitor`
+            - `#elements_with_events` -- Optionally limits event inheritance.
+- `Rest::Server`
+    - Added `/scans/:id/report.afr`.
+    - Added error handling for when trying to connect to killed instances.
+- `Support`
+    - `Database` -- Updated to compress disk data.
+- `Parser` -- Recode node data to UTF-8.
+- `Process`
+    - `Manager`
+        - `#spawn` -- Pass Arachni options via `ENV` rather than `ARGV`.
+- Checks
+    - Active
+        - `xss` -- Fixed proof data to return HTML.
+        - `csrf` -- Disabled for pages with DOM transitions due to FPs.
+    - Passive
+        - `x_frame_options` -- Ignore non-200 pages.
+        - `common_directories` -- Look for `.git`.
+- Plugins
+    - `proxy` -- Fixed error on login sequence recording.
+
+## 1.5.1 _(March 29, 2017)_
+
+- `config/write_paths.yml` -- Added configurable temporary directory.
+- `Parser`
+    - `#document` -- Updated to lazy parse the document.
+- `Browser`
+    - `Javascript`
+        - `DOMMonitor` -- Don't track `setInterval()`s since we're not using them.
+        - `TaintTracer`
+            - `add_trace_to_function()` -- Catch and return on error.
+- Path extractors
+    - `scripts` -- Fixed `nil` error.
+- Plugins
+    - `metrics` -- Fixed type error due to race condition.
+
 ## 1.5 _(January 31, 2017)_
 
 - Executables

data/Gemfile

@@ -1,6 +1,6 @@
 source 'https://rubygems.org'
 
-gem 'rake', '11.3.0'
+gem 'rake', '>= 12.3.3'
 gem 'pry'
 
 group :docs do
@@ -9,9 +9,7 @@ group :docs do
 end
 
 group :spec do
-    gem 'simplecov', require: false, group: :test
-
-    gem 'rspec', '2.99.0'
+    gem 'rspec'
     gem 'faker'
 end
 

data/LICENSE.md

@@ -1,6 +1,6 @@
 # License
 
-Copyright 2010-2017 [Sarosys LLC](http://www.sarosys.com).
+Copyright 2010-2022 [Ecsypno](http://www.ecsypno.com).
 
 ```
                     Arachni Public Source License

data/README.md

@@ -3,7 +3,7 @@
 <table>
     <tr>
         <th>Version</th>
-        <td>1.5</td>
+        <td>1.6.1</td>
     </tr>
     <tr>
         <th>Homepage</th>
@@ -38,7 +38,7 @@
     </tr>
     <tr>
         <th>Copyright</th>
-        <td>2010-2017 <a href="http://www.sarosys.com">Sarosys LLC</a></td>
+        <td>2010-2022 <a href="http://www.ecsypno.com">Ecsypno</a></td>
     </tr>
     <tr>
         <th>License</th>
@@ -112,27 +112,27 @@ you with its findings.
 
 ### General
 
- - Cookie-jar/cookie-string support.
- - Custom header support.
- - SSL support with fine-grained options.
- - User Agent spoofing.
- - Proxy support for SOCKS4, SOCKS4A, SOCKS5, HTTP/1.1 and HTTP/1.0.
- - Proxy authentication.
- - Site authentication (SSL-based, form-based, Cookie-Jar, Basic-Digest, NTLMv1, Kerberos and others).
- - Automatic log-out detection and re-login during the scan (when the initial
-    login was performed via the `autologin`, `login_script` or `proxy` plugins).
- - Custom 404 page detection.
- - UI abstraction:
+- Cookie-jar/cookie-string support.
+- Custom header support.
+- SSL support with fine-grained options.
+- User Agent spoofing.
+- Proxy support for SOCKS4, SOCKS4A, SOCKS5, HTTP/1.1 and HTTP/1.0.
+- Proxy authentication.
+- Site authentication (SSL-based, form-based, Cookie-Jar, Basic-Digest, NTLMv1, Kerberos and others).
+- Automatic log-out detection and re-login during the scan (when the initial
+  login was performed via the `autologin`, `login_script` or `proxy` plugins).
+- Custom 404 page detection.
+- UI abstraction:
     - [Command-line Interface](https://github.com/Arachni/arachni/wiki/Executables).
     - [Web User Interface](https://github.com/Arachni/arachni-ui-web).
- - Pause/resume functionality.
- - Hibernation support -- Suspend to and restore from disk.
- - High performance asynchronous HTTP requests.
+- Pause/resume functionality.
+- Hibernation support -- Suspend to and restore from disk.
+- High performance asynchronous HTTP requests.
     - With adjustable concurrency.
     - With the ability to auto-detect server health and adjust its concurrency
-        automatically.
- - Support for custom default input values, using pairs of patterns (to be matched
-    against input names) and values to be used to fill in matching inputs.
+      automatically.
+- Support for custom default input values, using pairs of patterns (to be matched
+  against input names) and values to be used to fill in matching inputs.
 
 ### Integrated browser environment
 
@@ -155,27 +155,27 @@ with a great deal of information regarding the state of the page at the time.
 
 Relevant information include:
 
- - Page DOM, as HTML code.
-     - With a list of DOM transitions required to restore the state of the
-         page to the one at the time it was logged.
- - Original DOM (i.e. prior to the action that caused the page to be logged),
-     as HTML code.
-     - With a list of DOM transitions.
- - Data-flow sinks -- Each sink is a JS method which received a tainted argument.
-     - Parent object of the method (ex.: `DOMWindow`).
-     - Method signature (ex.: `decodeURIComponent()`).
-     - Arguments list.
-         - With the identified taint located recursively in the included objects.
-     - Method source code.
-     - JS stacktrace.
- - Execution flow sinks -- Each sink is a successfully executed JS payload,
-     as injected by the security checks.
-     - Includes a JS stacktrace.
- - JavaScript stack-traces include:
-     - Method names.
-     - Method locations.
-     - Method source codes.
-     - Argument lists.
+- Page DOM, as HTML code.
+    - With a list of DOM transitions required to restore the state of the
+      page to the one at the time it was logged.
+- Original DOM (i.e. prior to the action that caused the page to be logged),
+  as HTML code.
+    - With a list of DOM transitions.
+- Data-flow sinks -- Each sink is a JS method which received a tainted argument.
+    - Parent object of the method (ex.: `DOMWindow`).
+    - Method signature (ex.: `decodeURIComponent()`).
+    - Arguments list.
+        - With the identified taint located recursively in the included objects.
+    - Method source code.
+    - JS stacktrace.
+- Execution flow sinks -- Each sink is a successfully executed JS payload,
+  as injected by the security checks.
+    - Includes a JS stacktrace.
+- JavaScript stack-traces include:
+    - Method names.
+    - Method locations.
+    - Method source codes.
+    - Argument lists.
 
 In essence, you have access to roughly the same information that your favorite
 debugger (for example, FireBug) would provide, as if you had set a breakpoint to
@@ -189,15 +189,15 @@ consuming in a high-performance fashion.
 
 Configuration options include:
 
- - Adjustable pool-size, i.e. the amount of browser workers to utilize.
- - Timeout for each job.
- - Worker TTL counted in jobs -- Workers which exceed the TTL have their browser
-     process respawned.
- - Ability to disable loading images.
- - Adjustable screen width and height.
-     - Can be used to analyze responsive and mobile applications.
- - Ability to wait until certain elements appear in the page.
- - Configurable local storage data.
+- Adjustable pool-size, i.e. the amount of browser workers to utilize.
+- Timeout for each job.
+- Worker TTL counted in jobs -- Workers which exceed the TTL have their browser
+  process respawned.
+- Ability to disable loading images.
+- Adjustable screen width and height.
+    - Can be used to analyze responsive and mobile applications.
+- Ability to wait until certain elements appear in the page.
+- Configurable local storage data.
 
 ### Coverage
 
@@ -212,27 +212,28 @@ order to provide coverage for a full set of possible scenarios.
 By inspecting all possible pages and their states (when using client-side code)
 Arachni is able to extract and audit the following elements and their inputs:
 
- - Forms
+- Forms
     - Along with ones that require interaction via a real browser due to DOM events.
- - User-interface Forms
+- User-interface Forms
     - Input and button groups which don't belong to an HTML `<form>` element but
-        are instead associated via JS code.
- - User-interface Inputs
+      are instead associated via JS code.
+- User-interface Inputs
     - Orphan `<input>` elements with associated DOM events.
- - Links
+- Links
     - Along with ones that have client-side parameters in their fragment, i.e.:
-        `http://example.com/#/?param=val&param2=val2`
+      `http://example.com/#/?param=val&param2=val2`
     - With support for rewrite rules.
- - LinkTemplates -- Allowing for extraction of arbitrary inputs from generic paths,
-    based on user-supplied templates -- useful when rewrite rules are not available.
+- LinkTemplates -- Allowing for extraction of arbitrary inputs from generic paths,
+  based on user-supplied templates -- useful when rewrite rules are not available.
     - Along with ones that have client-side parameters in their URL fragments, i.e.:
-            `http://example.com/#/param/val/param2/val2`
- - Cookies
- - Headers
- - Generic client-side elements which have associated DOM events.
- - AJAX-request parameters.
- - JSON request data.
- - XML request data.
+      `http://example.com/#/param/val/param2/val2`
+- Cookies
+    - Also supports nested cookies, containing key-value pairs inside individual cookies.
+- Headers
+- Generic client-side elements which have associated DOM events.
+- AJAX-request parameters.
+- JSON request data.
+- XML request data.
 
 ### Open [distributed architecture](https://github.com/Arachni/arachni/wiki/Distributed-components)
 
@@ -246,7 +247,7 @@ Both approaches allow you to:
 
 - Remotely monitor and manage scans.
 - Perform multiple scans at the same time -- Each scan is compartmentalized to
-    its own OS process to take advantage of:
+  its own OS process to take advantage of:
     - Multi-core/SMP architectures.
     - OS-level scheduling/restrictions.
     - Sandboxed failure propagation.
@@ -260,51 +261,51 @@ Both approaches allow you to:
     - Uses JSON to format messages.
 - Stateful scan monitoring.
     - Unique sessions automatically only receive updates when polling for progress,
-        rather than full data.
+      rather than full data.
 
 #### [RPC API](https://github.com/Arachni/arachni/wiki/RPC-API)
 
 - High-performance/low-bandwidth [communication protocol](https://github.com/Arachni/arachni-rpc).
     - `MessagePack` serialization for performance, efficiency and ease of
-        integration with 3rd party systems.
+      integration with 3rd party systems.
 - Grid:
     - Self-healing.
     - Scale up/down by hot-plugging/hot-unplugging nodes.
         - Can scale up infinitely by adding nodes to increase scan capacity.
     - _(Always-on)_ Load-balancing -- All Instances are automatically provided
-        by the least burdened Grid member.
+      by the least burdened Grid member.
         - With optional per-scan opt-out/override.
     - _(Optional)_ High-Performance mode -- Combines the resources of
-        multiple nodes to perform multi-Instance scans.
+      multiple nodes to perform multi-Instance scans.
         - Enabled on a per-scan basis.
 
 ### Scope configuration
 
- - Filters for redundant pages like galleries, catalogs, etc. based on regular
-    expressions and counters.
+- Filters for redundant pages like galleries, catalogs, etc. based on regular
+  expressions and counters.
     - Can optionally detect and ignore redundant pages automatically.
- - URL exclusion filters using regular expressions.
- - Page exclusion filters based on content, using regular expressions.
- - URL inclusion filters using regular expressions.
- - Can be forced to only follow HTTPS paths and not downgrade to HTTP.
- - Can optionally follow subdomains.
- - Adjustable page count limit.
- - Adjustable redirect limit.
- - Adjustable directory depth limit.
- - Adjustable DOM depth limit.
- - Adjustment using URL-rewrite rules.
- - Can read paths from multiple user supplied files (to both restrict and extend
-    the scope).
+- URL exclusion filters using regular expressions.
+- Page exclusion filters based on content, using regular expressions.
+- URL inclusion filters using regular expressions.
+- Can be forced to only follow HTTPS paths and not downgrade to HTTP.
+- Can optionally follow subdomains.
+- Adjustable page count limit.
+- Adjustable redirect limit.
+- Adjustable directory depth limit.
+- Adjustable DOM depth limit.
+- Adjustment using URL-rewrite rules.
+- Can read paths from multiple user supplied files (to both restrict and extend
+  the scope).
 
 ### Audit
 
- - Can audit:
+- Can audit:
     - Forms
         - Can automatically refresh nonce tokens.
         - Can submit them via the integrated browser environment.
-     - User-interface Forms
+    - User-interface Forms
         - Input and button groups which don't belong to an HTML `<form>` element
-            but are instead associated via JS code.
+          but are instead associated via JS code.
     - User-interface Inputs
         - Orphan `<input>` elements with associated DOM events.
     - Links
@@ -317,13 +318,13 @@ Both approaches allow you to:
     - Generic client-side DOM elements.
     - JSON request data.
     - XML request data.
- - Can ignore binary/non-text pages.
- - Can audit elements using both `GET` and `POST` HTTP methods.
- - Can inject both raw and HTTP encoded payloads.
- - Can submit all links and forms of the page along with the cookie
-    permutations to provide extensive cookie-audit coverage.
- - Can exclude specific input vectors by name.
- - Can include specific input vectors by name.
+- Can ignore binary/non-text pages.
+- Can audit elements using both `GET` and `POST` HTTP methods.
+- Can inject both raw and HTTP encoded payloads.
+- Can submit all links and forms of the page along with the cookie
+  permutations to provide extensive cookie-audit coverage.
+- Can exclude specific input vectors by name.
+- Can include specific input vectors by name.
 
 ### Components
 
@@ -514,7 +515,7 @@ Passive checks look for the existence of files, folders and signatures.
 
 - Standard output
 - [HTML](http://www.arachni-scanner.com/reports/report.html/)
-    ([zip](http://www.arachni-scanner.com/reports/report.html.zip)) (`html`).
+  ([zip](http://www.arachni-scanner.com/reports/report.html.zip)) (`html`).
 - [XML](http://www.arachni-scanner.com/reports/report.xml) (`xml`).
 - [Text](http://www.arachni-scanner.com/reports/report.txt) (`text`).
 - [JSON](http://www.arachni-scanner.com/reports/report.json) (`json`)
@@ -529,32 +530,32 @@ Plugins add extra functionality to the system in a modular fashion, this way the
 core remains lean and makes it easy for anyone to add arbitrary functionality.
 
 - Passive Proxy  (`proxy`) -- Analyzes requests and responses between the web app and
-    the browser assisting in AJAX audits, logging-in and/or restricting the scope of the audit.
+  the browser assisting in AJAX audits, logging-in and/or restricting the scope of the audit.
 - Form based login (`autologin`).
 - Script based login (`login_script`).
 - Dictionary attacker for HTTP Auth (`http_dicattack`).
 - Dictionary attacker for form based authentication (`form_dicattack`).
 - Cookie collector (`cookie_collector`) -- Keeps track of cookies while establishing a timeline of changes.
 - WAF (Web Application Firewall) Detector (`waf_detector`) -- Establishes a baseline of
-    normal behavior and uses rDiff analysis to determine if malicious inputs cause any behavioral changes.
+  normal behavior and uses rDiff analysis to determine if malicious inputs cause any behavioral changes.
 - BeepNotify (`beep_notify`) -- Beeps when the scan finishes.
 - EmailNotify (`email_notify`) -- Sends a notification (and optionally a report) over SMTP at
-    the end of the scan.
+  the end of the scan.
 - VectorFeed (`vector_feed`) -- Reads in vector data from which it creates elements to be
-    audited. Can be used to perform extremely specialized/narrow audits on a per vector/element basis.
-    Useful for unit-testing or a gazillion other things.
+  audited. Can be used to perform extremely specialized/narrow audits on a per vector/element basis.
+  Useful for unit-testing or a gazillion other things.
 - Script (`script`) -- Loads and runs an external Ruby script under the scope of a plugin,
-    used for debugging and general hackery.
+  used for debugging and general hackery.
 - Uncommon headers (`uncommon_headers`) -- Logs uncommon headers.
 - Content-types (`content_types`) -- Logs content-types of server responses aiding in the
-    identification of interesting (possibly leaked) files.
+  identification of interesting (possibly leaked) files.
 - Vector collector (`vector_collector`) -- Collects information about all seen input vectors
-    which are within the scan scope.
+  which are within the scan scope.
 - Headers collector (`headers_collector`) -- Collects response headers based on specified criteria.
 - Exec (`exec`) -- Calls external executables at different scan stages.
 - Metrics (`metrics`) -- Captures metrics about multiple aspects of the scan and the web application.
 - Restrict to DOM state (`restrict_to_dom_state`) -- Restricts the audit to a single page's DOM
-    state, based on a URL fragment.
+  state, based on a URL fragment.
 - Webhook notify (`webhook_notify`) -- Sends a webhook payload over HTTP at the end of the scan.
 - Rate limiter (`rate_limiter`) -- Rate limits HTTP requests.
 - Page dump (`page_dump`) -- Dumps page data to disk as YAML.
@@ -564,7 +565,7 @@ core remains lean and makes it easy for anyone to add arbitrary functionality.
 Default plugins will run for every scan and are placed under `/plugins/defaults/`.
 
 - AutoThrottle (`autothrottle`) -- Dynamically adjusts HTTP throughput during the scan for
-    maximum bandwidth utilization.
+  maximum bandwidth utilization.
 - Healthmap (`healthmap`) -- Generates sitemap showing the health of each crawled/audited URL
 
 ###### Meta
@@ -573,12 +574,12 @@ Plugins under `/plugins/defaults/meta/` perform analysis on the scan results
 to determine trustworthiness or just add context information or general insights.
 
 - TimingAttacks (`timing_attacks`) -- Provides a notice for issues uncovered by timing attacks
-    when the affected audited pages returned unusually high response times to begin with.
-    It also points out the danger of DoS attacks against pages that perform heavy-duty processing.
+  when the affected audited pages returned unusually high response times to begin with.
+  It also points out the danger of DoS attacks against pages that perform heavy-duty processing.
 - Discovery (`discovery`) -- Performs anomaly detection on issues logged by discovery
-    checks and warns of the possibility of false positives where applicable.
+  checks and warns of the possibility of false positives where applicable.
 - Uniformity (`uniformity`) -- Reports inputs that are uniformly vulnerable across a number
-    of pages hinting to the lack of a central point of input sanitization.
+  of pages hinting to the lack of a central point of input sanitization.
 
 ### Trainer subsystem
 
@@ -611,7 +612,7 @@ You can run `rake spec` to run **all** specs or you can run them selectively usi
 **Please be warned**, the core specs will require a beast of a machine due to the
 necessity to test the Grid/multi-Instance features of the system.
 
-**Note**: _The check specs will take about 90 minutes due to the timing-attack tests._
+**Note**: _The check specs will take many hours to complete due to the timing-attack tests._
 
 ## Bug reports/Feature requests
 
@@ -628,10 +629,10 @@ need to follow in order to contribute code:
 
 * Fork the project.
 * Start a feature branch based on the [experimental](https://github.com/Arachni/arachni/tree/experimental)
-    branch (`git checkout -b <feature-name> experimental`).
+  branch (`git checkout -b <feature-name> experimental`).
 * Add specs for your code.
 * Run the spec suite to make sure you didn't break anything (`rake spec:core`
-    for the core libs or `rake spec` for everything).
+  for the core libs or `rake spec` for everything).
 * Commit and push your changes.
 * Issue a pull request and wait for your code to be reviewed.
 

data/Rakefile

@@ -1,5 +1,5 @@
 =begin
-    Copyright 2010-2017 Sarosys LLC <http://www.sarosys.com>
+    Copyright 2010-2022 Ecsypno <http://www.ecsypno.com>
 
     This file is part of the Arachni Framework project and is subject to
     redistribution and commercial restrictions. Please see the Arachni Framework
@@ -197,7 +197,6 @@ end
 
 desc 'Generate docs.'
 task :docs do
-
     outdir = "../arachni-docs"
     sh "rm -rf #{outdir}"
     sh "mkdir -p #{outdir}"
@@ -207,47 +206,6 @@ task :docs do
     sh "rm -rf .yardoc"
 end
 
-desc 'Generate graphics.'
-task :gfx do
-
-    outdir = 'gfx/compiled'
-    srcdir = 'gfx/source'
-
-    sh 'mkdir -p ~/.fonts'
-    sh 'cp gfx/font/Beneath_the_Surface.ttf ~/.fonts'
-
-    Dir.glob( "#{srcdir}/*.svg" ).each do |src|
-        sh "inkscape #{src} --export-png=#{outdir}/#{File.basename( src, '.svg' )}.png"
-    end
-
-    cp "#{outdir}/icon.png", "#{outdir}/favicon.ico"
-
-    sh 'rm -f ~/.fonts/Beneath_the_Surface.ttf'
-end
-
-#
-# Simple profiler using perftools[1].
-#
-# To install perftools for Ruby:
-#   gem install perftools.rb
-#
-# [1] https://github.com/tmm1/perftools.rb
-#
-desc 'Profile Arachni.'
-task :profile do
-
-    if !Gem::Specification.find_all_by_name( 'perftools.rb' ).empty?
-        sh "CPUPROFILE_FREQUENCY=500 CPUPROFILE=/tmp/profile.dat " +
-               "RUBYOPT=\"-r`gem which perftools | tail -1`\" " +
-               " ./bin/arachni http://demo.testfire.net && " +
-               "pprof.rb --gif /tmp/profile.dat > profile.gif"
-    else
-        puts 'If you want to run the profiler please install perftools.rb first:'
-        puts '  gem install perftools.rb'
-    end
-
-end
-
 desc 'Remove reporter and log files.'
 task :clean do
     files = %w(error.log *.afr *.afs *.yaml *.json *.marshal *.gem pkg/*.gem