selenium_tor 1.3.0 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fcf1656f54ce23992e788e76b1d6342157b6ba3996ec95f04c955a2857e9b2be
-  data.tar.gz: a9ef924f85fdd1370638ff3c12f9009ec1b2ab1b65b5a872bb8bd3d3e3e6e4f5
+  metadata.gz: defca2c98d6db63f24a6b4dda2a2ade30ce248af24fe670e4237cd3feda2d2eb
+  data.tar.gz: a976b592807e153f0aaabff92a02e22f55bf900ea35b7435ba66f305990e8b6b
 SHA512:
-  metadata.gz: 68018127837d8e59dd60647d9a0d60da4af363da8fad7104ac6ed52bf15844d7ead4d4cbc81f1b2403e64ad2fdda55f488ce725ca71b23c618686fb3ed5dbde3
-  data.tar.gz: 3a6bd3db60f2a9de62e232774edabb11e4ccd7bb6e48158c4b6709339f8c964e56f4817f78d471d8ae57a970fec25b325a76a4c3779a3ee67aee8bdc2ad5250f
+  metadata.gz: 76f9832dc38aac26dce3591d0c624f9f57dab9aca02ef732dcf6beb13ffbc3c0ab090964ee12cacc2b8bef7b16981f8ba1febfe5733eb3125c8e7899cf99d95a
+  data.tar.gz: 9bb0c63ff4cfeaf847d8a3adaaac562ef9c74783c4629976de3d54b9014d2316b28828276e1862e8e0be19c6d0625c897c04cee3dc17dcf5d72ede23223461b9

data/CHANGELOG.md

@@ -1,10 +1,32 @@
 ## master (unreleased)
 
-## [1.3.0] - 2024-08-05
+## [1.5.0] - 2024-08-31
+
+### Bug fixes
+
+* [#15](https://gitlab.com/matzfan/selenium-tor/-/issues/15)
+* [#16](https://gitlab.com/matzfan/selenium-tor/-/issues/16)
+* [#17](https://gitlab.com/matzfan/selenium-tor/-/issues/17)
+
+### New features
+
+* add ability to patch libxul.so to avoid webdriver fingerprint detection
+
+## [1.4.0] - 2024-08-07
+
+### Breaking changes
+
+* only :socks_port and :control_port symbols are valid in :tor_opts
+
+### New features
+
+* add ability to set tor bootstrap timeout with tor_opts
 
 ### Bug fixes
 
-* [#14](https://gitlab.com/matzfan/selenium-tor/-/issues/14)
+* [#8](https://gitlab.com/matzfan/selenium-tor/-/issues/8)
+
+## [1.3.0] - 2024-08-05
 
 ### New features
 

data/README.md

@@ -5,8 +5,14 @@
 
 A Selenium extension for Tor Browser.
 ```ruby
-@driver = Selenium::WebDriver.for :tor
-@driver.quit
+options = Selenium::WebDriver::Tor::Options.new
+@driver = Selenium::WebDriver.for :tor, options: options
+```
+Once you have a driver instance:
+```ruby
+@driver.get 'https://check.torproject.org'
+@driver.title
+# => Congratulations. This browser is configured to use Tor.
 ```
 
 ## \_why?
@@ -15,14 +21,6 @@ I can use Firefox with Selenium and set a SOCKS proxy to use the Tor network, so
 
 The above approach will hide your IP, but there is a good chance your browser's unique or near-unique fingerprint may be logged by site owners. Subsequent visits could identify you. A primary aim of this project is to enable Selenium to leverage Tor Browser's unique anonymity characteristics - in particular its resistance to browser fingerprinting. The aim is to ensure Selenium Tor site visits leave an identical fingerprint to the thousands of regular Tor Browser users.
 
-## Known issues
-
-Known issues are recorded [here](https://gitlab.com/matzfan/selenium-tor/-/issues).
-
-### Fingerprinting
-
-The gem uses Xvfb to allow Tor Browser to be manipulated headlessly. Xvfb in turn uses `llvmpipe` software acceleration which may lead to issues with WebGL fingerprinting - see [issue #7](https://gitlab.com/matzfan/selenium-tor/-/issues/7). If this is a problem we recommend using [VirtualGL](https://www.virtualgl.org) to force Xvfb to use your hardware driver instead. This is done by prepending your executable code with the command `vglrun`. For an example see the section below on testing. VirtualGL can be installed from a [package repo](https://virtualgl.org/Downloads/YUM). 
-
 ## Installation
 
 Install the gem and add to the application's Gemfile by executing:
@@ -33,27 +31,54 @@ If bundler is not being used to manage dependencies, install the gem by executin
 
     $ gem install selenium_tor
 
+## Dependencies and configuration
+
+[Tor Browser](https://www.torproject.org/download).
+
+The shared libraries Tor Bowser requires will be available if you have Firefox or Firefox ESR installed on your system. If not, do `sudo apt install firefox-esr` or the equivalent for your package manager.
+
+As with Firefox browser, `geckodriver` needs to be installed and in your PATH.
+
+The gem needs to know the location of the Tor Browser Bundle (TBB). The Tor Browser download package archive must be extracted and the root TBB directory (named "tor-browser") placed somewhere on your system. By default it is assumed to be in the current user's HOME directory. An alternative location can be set via the env var `TOR_BROWSER_ROOT_DIR` - e.g. `export TOR_BROWSER_ROOT_DIR=/home/<user>/Downloads`. The Tor Browser binary location is *automatically set* by reference to this directory, so there is no need to do this:
+```ruby
+options = Selenium::WebDriver::Tor::Options.new
+options.binary = '/some/path/to/tor_firefox_binary' # UNNECESSARY
+```
+
+The location of the TBB is not expected to change during code execution.
+
+Tor Selenium is tested on **Linux only** right now.
+
+### Patching libxul.so
+
+A configuration option to patch the libxul.so packaged with Tor Browser is provided. This has the effect of preventing websites from fingerprinting Selenium-driven Tor Browser via a JS call to `navigator.webdriver`:
+```ruby
+Tor::Driver.libxul_patched? # is Tor Browser's libxul.so patched?
+# => false
+Tor::Driver.patch_libxul # patch libxul.so
+Tor::Driver.libxul_patched?
+# => true
+Tor::Driver.unpatch_libxul # revert the patch and restore the libxul.so packaged with Tor Browser
+```
+The patcher uses the `[bsdiff](https://www.daemonology.net/bsdiff)` package, which must be installed on your system.
+
 ## Usage
 ```ruby
 require 'selenium_tor'
 ```
 Tor Browser is based on Firefox, so for usage please read the Selenium [docs](https://www.selenium.dev/documentation/webdriver/browsers/firefox) for Firefox browser.
 
-A driver is instantiated like this:
-```ruby
-options = Selenium::WebDriver::Tor::Options.new
-@driver = Selenium::WebDriver.for :tor, options: options
+If the tor network is inaccessible for any reason a `Selenium::WebDriver::Error::TimeoutError` will result.
 
-@driver.get 'https://check.torproject.org'
-@driver.title
-# => Congratulations. This browser is configured to use Tor.
-@driver.quit
-```
-If the network is inaccessible for any reason a `TorNetworkError` will result.
+A separate tor process is used for each driver. Failure to call `Driver#quit` after code execution may leave orphaned tor processes.
+
+### Tor options
+
+In addition to the regular Firefox options, a :tor_opts key may be passed to an instance of `Tor::Options` with a hash of tor options. All valid tor options are recognized - see `man tor`. For convenience, "SocksPort" and "ControlPort" options may be set using snake_case symbols - i.e. :socks_port and :control_port. Additionally, a :tor_opts timeout value may be set with the :timeout key. This overrides the default time allowed for the tor process to bootstrap (10 seconds).
 
 ### Multiple driver instances
 
-Running multiple `tor` processes requires that each uses different ports for SocksPort and ControlPort. These and other valid tor options can be passed using the `:tor_opts` key. Recognized options are snake_case equivalents of the camel case options reconized by tor. For a list see `man tor`. An example using the [Parallel gem](https://rubygems.org/gems/parallel):
+Running multiple tor processes requires that each uses different ports for SocksPort (and ControlPort, if used). These must be passed using the `:tor_opts` key. An example using the [Parallel gem](https://rubygems.org/gems/parallel):
 ```ruby
 require 'parallel'
 
@@ -77,7 +102,7 @@ end
 ```
 ### Tor Browser specific functionality
 
-You can get and set the secuirty level (shield icon in TB) as follows. 4 is 'Standard', 2 is 'Safer', 1 is 'Safest'.
+You can get and set the security level (shield icon in TB) as follows. 4 is 'Standard', 2 is 'Safer', 1 is 'Safest'.
 ```ruby
 @driver.security_level
 # => 4 - the default value
@@ -109,28 +134,19 @@ A number of constants are set during driver initialization based upon values fou
 Tor::TBB_DIR # path to the TBB root directory
 Tor::TBB_BROWSER_DIR # path to the 'Browser' directory in the above
 Tor::TBB_BINARY_PATH # path to the firefox binary
+Tor::TBB_TOR_BINARY_PATH # path to the bundled tor binary
 Tor::TBB_PROFILE_DIR # path to the default profile directory
 Tor::TBB_EXTENSIONS_DIR # path to the 'extensions' directory in the above
 Tor::TBB_VERSION # the version installed, e.g. "13.0.1", note: driver.capabilities.browser_version returns the Firefox version Tor Browser is based on 
 ```
 
-## Dependencies and configuration
-
-[Tor Browser](https://www.torproject.org/download).
-
-The shared libraries Tor Bowser requires will be available if you have Firefox or Firefox ESR installed on your system. If not, do `sudo apt install firefox-esr` or similar.
-
-As with Firefox browser, `geckodriver` needs to be installed and in your PATH.
+## Known issues
 
-The gem needs to know the location of the Tor Browser Bundle (TBB). The Tor Browser download package archive must be extracted and the root TBB directory (named "tor-browser") placed somewhere on your system. By default it is assumed to be in the current user's HOME directory. An alternative location can be set via the env var `TOR_BROWSER_ROOT_DIR` - e.g. `export TOR_BROWSER_ROOT_DIR=/home/<user>/Downloads`. The Tor Browser binary location is *automatically set* by reference to this directory, so there is no need to do this:
-```ruby
-options = Selenium::WebDriver::Tor::Options.new
-options.binary = '/some/path/to/tor_firefox_binary' # UNNECESSARY
-```
+Known issues are recorded [here](https://gitlab.com/matzfan/selenium-tor/-/issues).
 
-The location of the TBB is not expected to change during code execution.
+### Fingerprinting
 
-Tor Selenium is tested on **Linux only** right now.
+The gem uses Xvfb to allow Tor Browser to be manipulated headlessly. Xvfb in turn uses `llvmpipe` software acceleration which may lead to issues with WebGL fingerprinting - see [issue #7](https://gitlab.com/matzfan/selenium-tor/-/issues/7). If this is a problem we recommend using [VirtualGL](https://www.virtualgl.org) to force Xvfb to use your hardware driver instead. This is done by prepending your executable code with the command `vglrun`. For an example see the section below on testing. VirtualGL can be installed from a [package repo](https://virtualgl.org/Downloads/YUM).
 
 ## Testing
 
@@ -140,7 +156,7 @@ Tests are run in the display set by the `DISPLAY` env var, usually :0 by default
 
     $ DISPLAY=:99 bundle exec rake
 
-If you find tests are failing with `TorNetworkError` and a timeout message, check you have no other Tor Browser processes running. The processes to look for are either "firefox.real" or "firefox-esr". The latter may be legitimate if you are also running Firefox browser. You may also want to check the Tor network is actually up, it isn't always..
+If you find driver instantiation failing with port bind failure error messages ( these include "Address already in use. Is Tor already running?") check you have no other tor processes running with conflicting ports. With timeout errors, you may also want to check the Tor network is actually up, it isn't always..
 
 If you wish run the `vglrun` WebGL fingerprint test install VirtualGL (see above, assumes you have the relevent drivers installed) and run the following command:
 

data/lib/tor/driver.rb

@@ -4,6 +4,7 @@ require 'open-uri'
 require_relative '../service'
 require_relative '../options'
 require_relative '../firefox_prefs'
+require_relative 'libxul_patchable'
 require_relative 'profile'
 require_relative 'tor_process'
 
@@ -15,6 +16,8 @@ module Selenium
 
       # tor driver
       class Driver < DelegateClass(DriverDelegate)
+        extend LibxulPatchable
+
         TORRC_PATH_PREF = 'extensions.torlauncher.torrc_path'
         TORRC = 'torrc'
         BROWSER_SECURITY_LEVEL_PREF = 'browser.security_level.security_slider'
@@ -69,12 +72,18 @@ module Selenium
         end
 
         def create_tor_process_and_start_tor(opts)
-          @tor_process = TorProcess.new(@data_dir, opts || {})
-          @tor_process.start_tor
+          timeout = opts.delete :timeout
+          @tor_process = tor_process(opts)
+          @tor_process.start_tor(timeout: timeout)
         rescue Tor::TorProcess::TorProcessError => e
+          instance_variable_get(:@instance)&.quit # avoids hanging firefox.real process
           raise Error::WebDriverError, e
         end
 
+        def tor_process(opts)
+          TorProcess.new(@data_dir, opts || {})
+        end
+
         def domain
           URI(current_url).host&.match(/[^\.]+\.\w+$/)
         end

data/lib/tor/libxul_patchable.rb

@@ -0,0 +1,97 @@
+# frozen_string_literal: true
+
+require 'digest'
+require 'json'
+require 'open-uri'
+require_relative 'options' # module needs access to consts therein defined
+
+module Selenium
+  module WebDriver
+    module Tor
+      # methods to patch/unpatch libxul.so
+      module LibxulPatchable
+        class LibxulPatchableError < StandardError; end
+
+        # PATCH_REPO = 'https://gitlab.com/matzfan/keymaster/-/releases' # TODO
+        PATCH_REPO = 'https://gitlab.com/matzfan/keymaster/-/raw/main/patches/tor-browser'
+        ISSUES_URL = 'https://gitlab.com/matzfan/keymaster/-/issues'
+
+        def libxul_patched?
+          libxul_checksum = checksum_for Tor::TBB_LIBXUL
+          return true if patched_libxul_checksum_for_version == libxul_checksum
+          return false if unpatched_libxul_checksum_for_version == libxul_checksum
+
+          raise LibxulPatchableError, 'Unrecognized libxul.so'
+        end
+
+        def patch_libxul
+          return if libxul_patched?
+
+          patchfile = fetch_patchfile
+          valid? patchfile
+          FileUtils.mv Tor::TBB_LIBXUL, libxul_original # rename libxul.so
+          execute_patch patchfile
+          FileUtils.cp libxul_patched, Tor::TBB_LIBXUL
+        rescue OpenURI::HTTPError
+          raise LibxulPatchableError, "No patch for #{Tor::TBB_VERSION}, please raise an issue at #{ISSUES_URL}"
+        end
+
+        def unpatch_libxul
+          return unless libxul_patched?
+
+          FileUtils.mv libxul_original, Tor::TBB_LIBXUL
+        end
+
+        private
+
+        def libxul_original
+          "#{Tor::TBB_LIBXUL}.ORIGINAL"
+        end
+
+        def libxul_patched
+          "#{Tor::TBB_LIBXUL}.PATCHED"
+        end
+
+        def valid?(patchfile)
+          msg = "Invalid patch for #{Tor::TBB_VERSION}, please raise an issue at #{ISSUES_URL}"
+          raise LibxulPatchableError, msg unless patch_checksum_for_version == checksum_for(patchfile)
+        end
+
+        def execute_patch(patchfile)
+          raise LibxulPatchableError, 'bspatch not found, please install bsdiff' if `which bspatch`.empty?
+
+          `bspatch #{libxul_original} #{libxul_patched} #{patchfile}`
+        end
+
+        def unpatched_libxul_checksum_for_version
+          YAML.load_file('libxul_checksums.yml')[Tor::TBB_VERSION]['unpatched_libxul']
+        end
+
+        def patched_libxul_checksum_for_version
+          YAML.load_file('libxul_checksums.yml')[Tor::TBB_VERSION]['patched_libxul']
+        end
+
+        def patch_checksum_for_version
+          YAML.load_file('libxul_checksums.yml')[Tor::TBB_VERSION]['patch_checksum']
+        end
+
+        def checksum_for(file)
+          Digest::SHA256.file(file).hexdigest
+        end
+
+        def fetch_patchfile
+          tmp = Tempfile.new.tap(&:binmode)
+          URI.parse(patch_url).open('rb') do |f|
+            tmp.write(f.read)
+            tmp.tap(&:flush)
+          end
+          tmp.path
+        end
+
+        def patch_url
+          File.join PATCH_REPO, "#{Tor::TBB_VERSION}.patch"
+        end
+      end
+    end
+  end
+end

data/lib/tor/options.rb

@@ -15,8 +15,10 @@ module Selenium
 
         Tor::TBB_DIR = ENV.fetch('TOR_BROWSER_ROOT_DIR', nil) || DEFAULT_TBB_DIR
         Tor::TBB_BROWSER_DIR = File.join Tor::TBB_DIR, 'Browser'
+        Tor::TBB_LIBXUL = File.join Tor::TBB_BROWSER_DIR, 'libxul.so'
         Tor::TBB_BINARY_PATH = File.join Tor::TBB_BROWSER_DIR, 'firefox'
         Tor::TBB_TOR_BINARY_PATH = File.join Tor::TBB_BROWSER_DIR, *%w[TorBrowser Tor tor]
+        Tor::TBB_TOR_DATA_DIR = File.join Tor::TBB_BROWSER_DIR, *%w[TorBrowser Data Tor]
         Tor::TBB_PROFILE_DIR = File.join Tor::TBB_BROWSER_DIR, *%w[TorBrowser Data Browser profile.default]
         Tor::TBB_EXTENSIONS_DIR = File.join Tor::TBB_PROFILE_DIR, 'extensions'
         Tor::TBB_VERSION = JSON.parse(File.read(File.join(Tor::TBB_BROWSER_DIR, 'tbb_version.json')))['version']

data/lib/tor/tor_process.rb

@@ -2,15 +2,12 @@
 
 require 'timeout'
 require_relative 'torrc'
-require_relative '../string_extensions'
 
 module Selenium
   module WebDriver
     module Tor
       # Respresentation of a tor process
       class TorProcess
-        include StringExtensions
-
         class TorProcessError < StandardError; end
 
         BOOTSTRAP_SUCCESS_REGEX = /Bootstrapped 100% \(done\): Done$/
@@ -28,12 +25,15 @@ module Selenium
           @config ||= setup_config # Hash to store torrc config
         end
 
-        def start_tor
+        def start_tor(timeout: 10)
           r, io = IO.pipe
-          pid = Process.spawn "#{TBB_TOR_BINARY_PATH} -f #{@torrc.path}", out: io, err: :out
+          pid = Process.spawn tor_command, out: io, err: :out
           io.close
-          errors = parse_tor_bootstrap_errors r
+          errors = parse_tor_bootstrap_errors_with_timeout(r, timeout: timeout)
           errors.empty? ? @pid = pid : raise(TorProcessError, "Tor failed to start with errors:\n\n#{errors}")
+        rescue Timeout::Error
+          Process.kill 'KILL', pid if pid
+          raise Error::TimeoutError, "Tor not bootstrapped after #{timeout} seconds"
         ensure
           r.close
         end
@@ -44,7 +44,7 @@ module Selenium
           @pid = nil
         end
 
-        # private
+        private
 
         def valid_data_dir?(dir)
           msg = 'data_dir must exist and be a dir'
@@ -52,9 +52,7 @@ module Selenium
         end
 
         def map_opts_to_torrc_keys(opts)
-          raise TorProcessError, 'Options keys must be snake case' unless opts.keys.map!(&:to_s).all?(&:snake_case?)
-
-          opts.transform_keys { |k| k.to_s.split('_').map(&:capitalize).join }
+          opts.transform_keys { |k| k.is_a?(String) ? k : k.to_s.split('_').map(&:capitalize).join }
         end
 
         def setup_config
@@ -62,12 +60,18 @@ module Selenium
           @torrc.config
         end
 
-        def parse_tor_bootstrap_errors(io)
+        def tor_command
+          "#{TBB_TOR_BINARY_PATH} -f #{@torrc.path}"
+        end
+
+        def parse_tor_bootstrap_errors_with_timeout(io, timeout:)
           lines = []
-          io.each_line do |line|
-            lines << line
-            break lines.join if line.match BOOTSTRAP_FAIL_REGEX
-            break '' if line.match BOOTSTRAP_SUCCESS_REGEX
+          Timeout.timeout timeout do
+            io.each_line do |line|
+              lines << line
+              break lines.join if line.match BOOTSTRAP_FAIL_REGEX
+              break '' if line.match BOOTSTRAP_SUCCESS_REGEX
+            end
           end
         end
       end

data/lib/tor/torrc.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-require_relative 'options' # for Tor::TBB_TOR_BINARY_PATH
+require_relative 'options'
 
 module Selenium
   module WebDriver
@@ -36,7 +36,13 @@ module Selenium
         private
 
         def write_default_config
-          write_to_config DEFAULT_PORTS.merge('DataDirectory' => @data_dir)
+          write_to_config DEFAULT_PORTS.merge(default_config)
+        end
+
+        def default_config
+          { 'DataDirectory' => @data_dir,
+            'GeoIPFile' => File.join(TBB_TOR_DATA_DIR, 'geoip'),
+            'GeoIPv6File' => File.join(TBB_TOR_DATA_DIR, 'geoip6') }
         end
 
         def validate_torrc_options(hash)

data/lib/tor/version.rb

@@ -3,7 +3,7 @@
 module Selenium
   module WebDriver
     module Tor
-      VERSION = '1.3.0'
+      VERSION = '1.5.0'
     end
   end
 end

data/libxul_checksums.yml

@@ -0,0 +1,13 @@
+---
+13.5.2:
+  unpatched_libxul: b2f8b3fd25466bd3ffa12fb34b422767b28e09577d5714d6d0c97f695dac15dc
+  patched_libxul: 7f761479c2d0307b97023455cfd7811b5749690302f54508b247720105dd0a3b
+  patch_checksum: 695f7534586c757781bd33a6dbc627cfe62d010edce20e5754fa5efff7aa80d0
+14.0a2:
+  unpatched_libxul: e0eda8db975fe315e9be32b3bf95ee62767c60ee34deac97a7163f8ab0480dc9
+  patched_libxul: ed05f9c383f4ecbce69d978aefc0625c44874193707ebe11166e714b3e2b4f64
+  patch_checksum: 0cf817f97bdbe51fba5ad61d566934c9ebdda8fce5540bc803aeca23e0ac3e8b
+14.0a3:
+  unpatched_libxul: f448fc180b547d622b969fe170028408f75493cf95be5e4c1180594daec2226b
+  patched_libxul: 337a4ef0e723069695294b37a4843126529c44e2bc4f90b1533b547100355db3
+  patch_checksum: 08dd8cab30c4416adf7ab6411e1346dce4a8b6f42e4c410adcd5bc63a71506e5

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: selenium_tor
 version: !ruby/object:Gem::Version
-  version: 1.3.0
+  version: 1.5.0
 platform: ruby
 authors:
 - MatzFan
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-08-05 00:00:00.000000000 Z
+date: 2024-08-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: selenium-webdriver
@@ -46,8 +46,8 @@ files:
 - lib/options.rb
 - lib/selenium_tor.rb
 - lib/service.rb
-- lib/string_extensions.rb
 - lib/tor/driver.rb
+- lib/tor/libxul_patchable.rb
 - lib/tor/options.rb
 - lib/tor/profile.rb
 - lib/tor/system_tor.rb
@@ -55,8 +55,10 @@ files:
 - lib/tor/tor_process.rb
 - lib/tor/torrc.rb
 - lib/tor/version.rb
+- libxul_checksums.yml
 - selenium_tor.gemspec
 - sig/tor.rbs
+- tmp/.placeholder
 homepage: https://gitlab.com/matzfan/selenium-tor
 licenses:
 - MIT

data/lib/string_extensions.rb

@@ -1,12 +0,0 @@
-# frozen_string_literal: true
-
-# patch String
-module StringExtensions
-  def snake_case?
-    match(/\A[a-z]+(_[a-z]+)?\z/)
-  end
-end
-
-class String
-  prepend StringExtensions
-end