middle_squid 1.0

data/README.md

@@ -0,0 +1,227 @@
+# MiddleSquid
+
+[![Gem Version](https://badge.fury.io/rb/middle_squid.svg)](http://badge.fury.io/rb/middle_squid)
+[![Build Status](https://travis-ci.org/cfillion/middle_squid.svg?branch=master)](https://travis-ci.org/cfillion/middle_squid)
+[![Dependency Status](https://gemnasium.com/cfillion/middle_squid.svg)](https://gemnasium.com/cfillion/middle_squid)
+[![Code Climate](https://codeclimate.com/github/cfillion/middle_squid/badges/gpa.svg)](https://codeclimate.com/github/cfillion/middle_squid)
+[![Coverage Status](https://img.shields.io/coveralls/cfillion/middle_squid.svg)](https://coveralls.io/r/cfillion/middle_squid?branch=master)
+
+MiddleSquid is a redirector, url mangler and webpage interceptor for the Squid HTTP proxy.
+
+**Features**
+
+- Configuration is done by writing a ruby script
+- Supports plain-text domains/urls blacklists
+- Can intercept and modify any HTTP request or response
+- Works with HTTPS
+    if [SslBump](http://wiki.squid-cache.org/Features/SslBump) is enabled.
+
+## Installation & Setup
+
+Assuming [Squid](http://www.squid-cache.org/) is installed and running as user 'proxy'.
+These instructions were written for [Arch Linux](https://www.archlinux.org/).
+Some adaptation to your favorite operating system may be necessary, at your
+discretion.
+
+**Dependencies:**
+
+- Squid version 3.4 or newer
+- Ruby version 2.1 or newer
+
+### Step 1: Set a home folder for user 'proxy'
+
+```sh
+sudo mkdir /home/proxy
+sudo chown proxy:proxy /home/proxy
+sudo usermod --home /home/proxy proxy
+```
+
+### Step 2: Install MiddleSquid
+
+```sh
+sudo su - proxy
+
+gem install middle_squid
+echo 'run lambda {|uri, extras| }' > middle_squid_config.rb
+
+exit
+```
+
+### Step 3: Create a launcher script
+
+Create the file `/usr/local/bin/middle_squid_wrapper.sh`:
+
+```sh 
+#!/bin/sh
+
+GEM_HOME=$(ruby -e 'puts Gem.user_dir')
+exec $GEM_HOME/bin/middle_squid $*
+```
+
+### Step 4: Setup Squid
+
+Add these lines to your `/etc/squid/squid.conf`:
+
+```squidconf
+url_rewrite_program /usr/bin/sh /usr/local/bin/middle_squid_wrapper.sh start -C /home/proxy/middle_squid_config.rb
+
+# required to fix HTTPS sites (if SslBump is enabled)
+acl fix_ssl_rewrite method GET
+acl fix_ssl_rewrite method POST
+url_rewrite_access allow fix_ssl_rewrite
+url_rewrite_access deny all
+```
+
+Finish with `sudo squid -k reconfigure`. Check `/var/log/squid/cache.log` for errors.
+
+## Configuration
+
+MiddleSquid is configured by the ruby script specified in the command line by the `-C` or `--config-file` argument.
+
+The script must call the `run` method:
+
+```ruby
+run lambda {|uri, extras|
+  # decide what to do with uri
+}
+```
+
+The argument must be an object that responds to the `call` method and taking two arguments:
+the URI to process and an array of extra data received from squid
+(see url_rewrite_extras in
+[squid's documentation](http://www.squid-cache.org/Doc/config/url_rewrite_extras/)).
+
+Write this in the file `/home/proxy/middle_squid_config.rb` we have created earlier:
+
+```ruby
+run lambda {|uri, extras|
+  redirect_to 'http://duckduckgo.com' if uri.host.end_with? 'google.com'
+}
+```
+
+Run `sudo squid -k reconfigure` again to restart all MiddleSquid processes.
+You should now be redirected to http://duckduckgo.com each time you visit
+Google under your Squid proxy.
+
+### Black Lists
+
+While it may be fun to redirect yourself to an alternate search engine,
+MiddleSquid is more useful at blocking annoying advertisements and tracking
+services that are constantly watching your whereabouts.
+
+MiddleSquid can scan any black list collection distributed in plain-text format
+and compatible with SquidGuard or Dansguardian, such as:
+
+- [Shalla's Blacklists](http://www.shallalist.de/) (free for personal use)
+- [URLBlackList.com](http://www.urlblacklist.com/) (commercial)
+
+Replace the previous configuration in `/home/proxy/middle_squid_config.rb`
+by this one:
+
+```ruby
+database '/home/proxy/blacklist.db'
+
+adv     = blacklist 'adv'
+tracker = blacklist 'tracker'
+
+run lambda {|uri, extras|
+  if adv.include? uri
+    redirect_to 'http://your.webserver/block_pages/advertising.html'
+  end
+
+  if tracker.include? uri
+    redirect_to 'http://your.webserver/block_pages/tracker.html'
+  end
+}
+```
+
+Next we have to download a blacklist and ask MiddleSquid to index its content
+in the database for fast access:
+
+```sh
+sudo su - proxy
+
+# Download Shalla's Blacklists
+wget "http://www.shallalist.de/Downloads/shallalist.tar.gz" -O shallalist.tar.gz
+tar xzf shallalist.tar.gz
+mv BL ShallaBlackList
+
+# Construct the blacklist database
+/usr/local/bin/middle_squid_wrapper.sh index ShallaBlackList -C /etc/squid/middle_squid.rb
+
+exit
+```
+
+The `index` command above may take a while to complete. Once it's done, re-run `squid -k reconfigure` and
+enjoy an internet without ads or tracking beacons.
+
+### Content Interception
+
+MiddleSquid can also intercept the client's requests and modify the data sent to the
+browser. Let's translate a few click-bait headlines on BuzzFeed
+(check out [Downworthy](http://downworthy.snipe.net/) while you are at it):
+
+```ruby
+CLICK_BAITS = {
+  'Literally' => 'Figuratively',
+  'Mind-Blowing' => 'Painfully Ordinary',
+  'Will Blow Your Mind' => 'Might Perhaps Mildly Entertain You For a Moment',
+  # ...
+}.freeze
+
+define_action :translate do |uri|
+  intercept {|req, res|
+    status, headers, body = download_like req, uri
+
+    content_type = headers['Content-Type'].to_s
+
+    if content_type.include? 'text/html'
+      CLICK_BAITS.each {|before, after|
+        body.gsub! before, after
+      }
+    end
+
+    [status, headers, body]
+  }
+end
+
+run lambda {|uri, extras|
+  if uri.host == 'www.buzzfeed.com'
+    translate uri
+  end
+}
+```
+
+Don't use this feature unless you have the permission from all your users to do so.
+This indeed constitutes a man-in-the-middle attack and should be used with
+moderation.
+
+## Documentation
+
+MiddleSquid's documentation is hosted at
+[http://rubydoc.info/gems/middle_squid/MiddleSquid](http://rubydoc.info/gems/middle_squid/MiddleSquid).
+
+- [Configuration syntax (DSL)](http://rubydoc.info/gems/middle_squid/MiddleSquid/Builder)
+- [List of predefined actions](http://rubydoc.info/gems/middle_squid/MiddleSquid/Actions)
+- [List of predefined helpers](http://rubydoc.info/gems/middle_squid/MiddleSquid/Helpers)
+- [Available adapters](http://rubydoc.info/gems/middle_squid/MiddleSquid/Adapters)
+
+## Changelog
+
+### v1.0 (2014-10-05)
+
+First public release.
+
+## Future Plans
+
+- Find out why HTTPS is not always working under Squid without the ACL hack.
+- Write new adapters for other proxies or softwares.
+
+## Contributing
+
+1. [Fork it](https://github.com/cfillion/middle_squid/fork)
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Test your changes (`rake`)
+4. Commit your changes (`git commit -am 'Add some feature'`)
+5. Push to the branch (`git push origin my-new-feature`)
+6. Create a new Pull Request

data/Rakefile

@@ -0,0 +1,7 @@
+require 'bundler/gem_tasks'
+require 'rake/testtask'
+
+task :default => [:test]
+Rake::TestTask.new do |t|
+  t.test_files = FileList['test/helper.rb', 'test/test_*.rb']
+end

data/bin/middle_squid

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+
+Signal.trap('INT') { abort }
+
+require 'middle_squid'
+
+MiddleSquid::CLI.start ARGV

data/lib/middle_squid/actions.rb

@@ -0,0 +1,77 @@
+module MiddleSquid::Actions
+  #
+  # @!group Predefined Actions
+  #
+
+  # Allow the request to pass through. This is the default action.
+  #
+  # @example Whitelist a domain
+  #   run lambda {|uri, extras|
+  #     accept if uri.host == 'github.com'
+  #   }
+  def accept
+    action :accept
+  end
+
+  # Redirect the browser to another URL.
+  #
+  # @example Redirect google.com to duckduckgo.com
+  #   run lambda {|uri, extras|
+  #     redirect_to "http://duckduckgo.com/#{uri.request_uri}" if uri.host == 'google.com'
+  #   }
+  # @param url [String] the new url
+  # @param status [Fixnum] HTTP status code (see http://tools.ietf.org/html/rfc7231#section-6.4)
+  def redirect_to(url, status: 301)
+    action :redirect, status: status, url: url
+  end
+
+  # Serve another page in place of the requested one.
+  # Avoid in favor of {#redirect_to} when possible.
+  #
+  # @example Block Google advertisements
+  #   run lambda {|uri, extras|
+  #     redirect_to 'http://webserver.lan/blocked.html' if uri.host == 'ads.google.com'
+  #   }
+  # @param url [String] the substitute url
+  def replace_by(url)
+    action :replace, url: url
+  end
+
+  # Hijack the request and generate a dynamic reply.
+  # This can be used to skip landing pages,
+  # change the behaviour of a website depending on the browser's headers or to
+  # generate an entire virtual website using your favorite Rack framework.
+  #
+  # The block is called inside a fiber.
+  # If the return value is a Rack triplet, it will be sent to the browser.
+  #
+  # @note
+  #   With great power comes great responsibility.
+  #   Please respect the privacy of your users.
+  # @example Hello World
+  #   run lambda {|uri, extras|
+  #     intercept {|req, res|
+  #       [200, {}, 'Hello World']
+  #     }
+  #   }
+  # @yieldparam req [Rack::Request] the browser request
+  # @yieldparam res [Thin::AsyncResponse] the response to send back
+  # @yieldreturn Rack triplet or anything else
+  # @see Helpers#download_like
+  def intercept(&block)
+    raise ArgumentError, 'no block given' unless block_given?
+
+    token = server.token_for block
+
+    replace_by "http://#{server.host}:#{server.port}/#{token}"
+  end
+
+  #
+  # @!endgroup
+  #
+
+  private
+  def action(name, options = {})
+    throw :action, [name, options]
+  end
+end

data/lib/middle_squid/adapter.rb

@@ -0,0 +1,54 @@
+module MiddleSquid
+  # Base class for MiddleSquid's adapters.
+  # Subclasses should call {#handle} when they have received and parsed a request.
+  #
+  # @abstract Subclass and override {#output} to implement a custom adapter.
+  class Adapter
+    # Returns whatever was passed to {Builder#run}.
+    #
+    # @return [#call]
+    attr_accessor :handler
+
+    # Returns a new instance of Adapter.
+    # Use {Builder#use} instead.
+    def initialize(options = {})
+      @options = options
+    end
+
+    # Execute the user handler (see {#handler}) and calls +#output+.
+    #
+    # @param url <String> string representation of the url to be processed
+    # @param extras <Array> extra data to pass to the user's handler
+    def handle(url, extras = [])
+      uri = MiddleSquid::URI.parse url
+      raise InvalidURIError, "invalid URL received: '#{url}'" if !uri || !uri.host
+
+      action, options = catch :action do
+        @handler.call uri, extras
+        throw :action, [:accept, {}]
+      end
+
+      output action, options
+    end
+
+    # Pass an action to an underlying software.
+    #
+    # accept::
+    #   (no options)
+    #
+    # redirect::
+    #   Options:
+    #   - +status+ [+Fixnum+]
+    #   - +url+ [+String+]
+    #
+    # replace::
+    #   Options:
+    #   - +url+ [+String+]
+    #
+    # @param action [Symbol]
+    # @param options [Hash]
+    def output(action, options)
+      raise NotImplementedError
+    end
+  end
+end

data/lib/middle_squid/adapters/squid.rb

@@ -0,0 +1,57 @@
+module MiddleSquid
+  # Adapter for the {http://www.squid-cache.org Squid HTTP Proxy}.
+  #
+  # *Options:*
+  #
+  # concurrency::
+  #   Whether to expect a channel ID from Squid.
+  #
+  #   Enable this option if the concurrency option is set to > 0 in Squid's
+  #   {http://www.squid-cache.org/Doc/config/url_rewrite_children/ url_rewrite_children} directive.
+  #
+  # Extra data is configured in Squid with the {http://www.squid-cache.org/Doc/config/url_rewrite_extras/ url_rewrite_extras} directive.
+  #
+  # @see http://wiki.squid-cache.org/Features/Redirectors
+  class Adapters::Squid < Adapter
+    def start
+      warn 'WARNING: STDOUT is a terminal. This command should be launched from squid.' if STDOUT.tty?
+
+      EM.open_keyboard Backends::Keyboard, method(:input)
+    end
+
+    def input(line)
+      parts = line.split
+
+      @chan_id = @options[:concurrency] ? parts.shift : nil
+      url, *extras = parts
+
+      extras.map! {|str| URI.unescape str }
+
+      handle url, extras
+    end
+
+    def output(action, options)
+      case action
+      when :accept
+        reply 'ERR'
+      when :redirect
+        reply 'OK', status: options[:status], url: options[:url]
+      when :replace
+        reply 'OK', :'rewrite-url' => options[:url]
+      else
+        raise Error, "unsupported action: #{action}"
+      end
+    end
+
+    private
+    def reply(result, **kv_pairs)
+      parts = []
+      parts << @chan_id if @chan_id
+      parts << result
+      parts.concat kv_pairs.map {|k,v| "#{k}=#{URI.escape v.to_s}" }
+
+      $stdout.puts parts.join("\x20")
+      $stdout.flush
+    end
+  end
+end

data/lib/middle_squid/backends/keyboard.rb

@@ -0,0 +1,31 @@
+module MiddleSquid::Backends
+  # Receives data from the standard input.
+  class Keyboard < EventMachine::Connection
+    # @param handler [#call] called when a full line has been received
+    def initialize(handler)
+      @buffer = []
+      @handler = handler
+    end
+
+    # @param char [String] single character
+    def receive_data(char)
+      case char
+      when "\x00"
+        EM.stop
+      when "\n"
+        line = @buffer.join
+        @buffer.clear
+
+        receive_line line
+      else
+        @buffer << char
+      end
+    end
+
+    # @param line [String] full line without the trailing linebreak
+    def receive_line(line)
+      # EventMachine sends ASCII-8BIT strings, somehow preventing the databases queries to match
+      @handler.call line.force_encoding(Encoding::UTF_8)
+    end
+  end
+end

data/lib/middle_squid/backends/thin.rb

@@ -0,0 +1,14 @@
+module MiddleSquid::Backends
+  # Exposes the signature of Thin's TCP socket.
+  #
+  # @example Extract the current host and port
+  #   sockname = EM.get_sockname @thin.backend.signature
+  #   @port, @host = Socket.unpack_sockaddr_in sockname
+  class Thin < Thin::Backends::TcpServer
+    attr_reader :signature
+
+    def initialize(host, port, options)
+      super host, port
+    end
+  end
+end

data/lib/middle_squid/blacklist.rb

@@ -0,0 +1,67 @@
+module MiddleSquid
+  # Use to query the blacklist database.
+  # URIs can be matched by hostname (see {#include_domain?}), path (see {#include_url?}) or both (see {#include?}).
+  #
+  # Instances of this class should be created using {Builder#blacklist Builder#blacklist}
+  # (otherwise they would not be seen by the "<code>middle_squid index</code>" command unless the "+--full+" flag is enabled).
+  class BlackList
+    include Database
+
+    # @return [String] the category passed to {#initialize}
+    attr_reader :category
+
+    # @return [Array<String>] the aliases passed to {#initialize}
+    attr_reader :aliases
+
+    # Returns a new instance of BlackList. Use {Builder#blacklist Builder#blacklist} instead.
+    # @param category [String]
+    # @param aliases  [Array<String>]
+    def initialize(category, aliases: [])
+      @category = category
+      @aliases = aliases
+    end
+
+    # Whether the blacklist category contains the URI's hostname or an upper-level domain.
+    #
+    # Rules to the <code>www</code> subdomain match any subdomains.
+    #
+    # @example Rule: sub.domain.com
+    #   Matches:
+    #   - http://sub.domain.com/...
+    #   - http://second.sub.domain.com/...
+    #   - http://infinite.level.of.sub.domain.com/...
+    # @param uri [URI] the URI to search
+    def include_domain?(uri)
+      !!db.get_first_value(
+        "SELECT 1 FROM domains WHERE category = ? AND ? LIKE '%' || host LIMIT 1",
+        [@category, uri.cleanhost]
+      )
+    end
+
+    # Whether the blacklist category contains the URI. Matches by partial domain (like {#include_domain?}) and path. The query string is ignored.
+    #
+    # Rules to index files (index.html, Default.aspx and friends) match the whole directory.
+    #
+    # @example Rule: domain.com/path
+    #   Matches:
+    #   - http://domain.com/path
+    #   - http://domain.com/dummy/../path
+    #   - http://domain.com/path/extra_path
+    #   - http://domain.com/path?query=string
+    # @param uri [URI] the URI to search
+    def include_url?(uri)
+      !!db.get_first_value(
+        "SELECT 1 FROM urls WHERE category = ? AND ? LIKE '%' || host AND ? LIKE path || '%' LIMIT 1",
+        [@category, uri.cleanhost, uri.cleanpath]
+      )
+    end
+
+    # Whether this blacklists contains the uri.
+    # Matches by domain and/or path.
+    #
+    # @param uri [URI] the uri to search
+    def include?(uri)
+      include_domain?(uri) || include_url?(uri)
+    end
+  end
+end