icaprb-filter 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 0837e275231b622dabdb5b873dc7e241bad263ee
+  data.tar.gz: 29a4e6006735b354efdd595ba16957323248451b
+SHA512:
+  metadata.gz: 3623f5d18b8c7d5699cb0f45e9cadd76b0fde8e4a5fbd08bfc151766151a5f91078136108fa465eb9eb3bb63d82ed70d1440911e93dc6d91852634f1367780b4
+  data.tar.gz: 5877b36f5a61248bc9a9aa4486d5b4df86a57b00cee9fabb0f5092cc44e8e81e1f793b4875325d58c40cc86fb3168751d1e70b19e34f57b2e0cd404b3eb317f2

data/.gitignore

@@ -0,0 +1,11 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+/html/
+/.idea/

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,4 @@
+language: ruby
+rvm:
+  - 2.3.0
+before_install: gem install bundler -v 1.11.2

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in icaprb-filter.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,25 @@
+Copyright (c) 2016, Fabian Franz
+Copyright (c) 2016, Markus Petz
+Copyright (c) 2016, Fabian Mayer
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice,
+   this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation and/or
+   other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
+SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
+TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
+EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+

data/README.md

@@ -0,0 +1,68 @@
+# ICAPrb::Filter
+
+ICAPrb::Filter is a content filter based on ICAPrb::Server. The goal is
+to develop a framework which allows to create any custom filtering of
+any content you would not like to have in your network. This could be
+ads, trackers, malware, adult content and so on.
+
+The way to defile rules should feel like configuring a firewall ruleset.
+
+Plug ins are automatically detected when they reside in the correct
+namespace and are available to use.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'icaprb-filter'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install icaprb-filter
+
+## Usage
+
+First of all, you will need a server which is serving your content
+filter.
+
+```ruby
+require "icaprb/server"
+require "icaprb/filter"
+
+s = ICAPServer.new
+# url: echo ist der Service - rechts ist der zu verwendende Service
+s.services['filter'] = ICAPrb::Server::FilterService.new('/path/to/filterconfig')
+s.run
+```
+You will need to replace the path by the path you use on your local
+machine.
+
+Next you will have to write your ruleset:
+
+```ruby
+request_filter do
+end
+
+response_filter do
+  block 'Block pages may contain samples',content_includes: ['sample']
+  block 'Block invalid domains', url_contains: ['.invalid']
+  block 'Block hashes of bad file', :check_hashes => ['~/hashes']
+end
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/fabianfrz/icaprb-filter.
+

data/README.rdoc

@@ -0,0 +1,22 @@
++ICAPrb::Filter+ is the filter component of +ICAPrb+.
+It aims to be able to filter content based on some matches as well as being able to use plug ins to modify content.
+
+Modify Content
+
+This feature is useful to remove some specific parts of your traffic.
+So you can remove some scripts (for example trackers and ads) based on theire content instead of the url as an URL can simply be changed.
+So the blocket content can be removed cleanly instead of having a whitespace somewhere on websites.
+The filter framework also supports actions like passing or blocking content.
+
+The supported methods are
+* pass:: passes the current state to the client
+         (in case the client supports 204 responses and the traffic has not been modified - it will be generated, otherwise the full data 
+         is sent to the client).
+* block:: sends a message that the content has been blocked
+* modify:: calls a plug in to modify the content depending on the return value the modified flag may be set which makes it impossible to
+           create a 204 response. For large files it is a good idea to have a pass rule generating 204 responses before any rule of this
+           type to reduce network traffic (and increase performance).
+* custom_action:: reserved for plug ins which should interact like a rule type (for example writing files out of the service for
+                  analysis. For example put it into a queue to test it inside a sandbox to detect malware)
+
+

data/Rakefile

@@ -0,0 +1,12 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require 'rdoc/task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec
+
+RDoc::Task.new do |rdoc|
+    rdoc.main = 'README.rdoc'
+    rdoc.rdoc_files.include('README.rdoc', 'lib/**/*.rb')
+end

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "icaprb/filter"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/bin/start_server.rb

@@ -0,0 +1,40 @@
+#
+require "bundler/setup"
+require "icaprb/server"
+require "icaprb/filter"
+require 'logger'
+include ICAPrb::Server
+
+trap('SIGINT') { exit! 0 }
+########################################################################
+#                  DIFFERENT WAYS TO RUN THE SERVER                    #
+########################################################################
+
+# normal socket
+s = ICAPServer.new
+# puts 'Server is running on port 1344. Press CTRL+C to exit...'
+
+# squid v4 variant
+#options = {secure: true,
+#          certificate: '../cert.pem',
+#          key: '../key.pem',
+#          tls_socket: true}
+#s = ICAPServer.new('localhost',11344,options)
+# puts 'Server is running on port 11344. Press CTRL+C to exit...'
+
+# rfc 3507 variant
+#options = {secure: true,
+#           certificate: '../cert.pem',
+#           key: '../key.pem',
+#           tls_socket: false}
+#s = ICAPServer.new('localhost',1344,options)
+puts 'Server is running on port 1344. Press CTRL+C to exit...'
+
+########################################################################
+s.logger.level = Logger::DEBUG
+# TIMEOUT:
+# timeout_in_seconds = 10
+# s.services['filter'] = ICAPrb::Server::FilterService.new('~/conf/conf.conf',timeout_in_seconds)
+s.services['filter'] = ICAPrb::Server::FilterService.new('~/conf/conf.conf')
+
+s.run

data/icaprb-filter.gemspec

@@ -0,0 +1,33 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'icaprb/filter/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'icaprb-filter'
+  spec.version       = ICAPrb::Filter::VERSION
+  spec.authors       = ['Fabian Franz']
+  spec.email         = ['fabian.franz@students.fh-hagenberg.at']
+
+  spec.summary       = %q{Filter framework for ICAPrb-server. Does not work standalone. This gem includes
+                          a framework to create a content filter which allows to filter content instead of only the
+                          urls. It is extensible by plug ins. For example it can be used to enforce your companies
+                          network usage policies at the proxy.}
+  spec.homepage      = 'https://github.com/fabianfrz/ICAPrb-Filter'
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = 'exe'
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+
+  spec.add_dependency 'nokogiri', '~> 1.6', '>= 1.6.7'
+  spec.add_dependency 'uirusu' # Virustotal public api
+  spec.add_dependency 'pdf-reader', '~> 1.4'
+  spec.add_dependency 'chunky_png'
+  spec.add_dependency 'icaprb-server'
+  spec.add_dependency 'rest-client'
+  # spec.add_dependency 'ruby-filemagic', '~> 0.7.1'
+  spec.add_development_dependency 'bundler', '~> 1.11'
+  spec.add_development_dependency 'rake'
+  spec.add_development_dependency 'rspec', '~> 3.0'
+end

data/lib/icaprb/filter.rb

@@ -0,0 +1,189 @@
+require "icaprb/filter/version"
+# All filter types
+require "icaprb/filter/solution"
+require "icaprb/filter/service"
+require 'logger'
+
+# ICAPrb
+module ICAPrb
+  # Filter module
+  module Filter
+    # Exception which is thrown when the configuration file is invalid
+    class ConfigurationLoadError < StandardError
+    end
+    # Response mod filters+plugins
+    @resp_mod = nil
+    # Temporary during reading
+    @temp_reps_mod = nil
+    # Request mod filters+plugins
+    @req_mod = nil
+    # Temporary during reading
+    @temp_req_mod = nil
+
+    # set the configuration mode to request filter so the rules will know that they are request filter rules.
+    # It takes a block which is executed while the filter_type is set up.
+    def self.request_filter
+      @filter_type = :request_mod
+      yield
+      @filter_type = nil
+    end
+
+    # set the configuration mode to response filter so the rules will know that they are response filter rules.
+    # It takes a block which is executed while the filter_type is set up.
+    def self.response_filter
+      @filter_type = :response_mod
+      yield
+      @filter_type = nil
+    end
+
+    # returns the filters
+    def self.get_filters
+      return @filters
+    end
+
+    # creates a new block rule with the parameters
+    # +description+:: description of the rule, so it can be logged and the admin can troubleshoot the ruleset.
+    # +action+:: do something
+    #
+    # raises a ConfigurationLoadError if it is called outside of request_filter or response_filter
+    def self.block(description, action)
+      puts "============================="
+      puts  "BLOCK: #{description}, ACTION NAME: #{action.keys.first}"
+      if @filter_type == :request_mod
+        puts "TYPE: REQUEST"
+      elsif @filter_type == :response_mod
+        puts  "TYPE: RESPONSE"
+      else
+        raise ConfigurationLoadError, "used param 'block' without a filter type"
+      end
+      # Check if action name exists and create it
+      found_name = false
+      @valid_filters.each { |filter|
+        if filter[:name].to_s == action.keys.first.to_s
+          found_name = true
+          new_filter = {:name => action.keys.first,
+                        :object => filter[:class].new(@filter_type, action[action.keys.first]),
+                        :description => description}
+          if @filter_type == :request_mod
+            @temp_req_mod << new_filter
+          elsif @filter_type == :response_mod
+            @temp_resp_mod << new_filter
+          end
+          break
+        end
+      }
+      unless found_name
+        raise ConfigurationLoadError, "Unknown filter type '" + action.keys.first.to_s + "' "
+      end
+    end
+    # Load a modifiying rule
+    # +description+:: Description of the rule
+    # +action+:: Name of the plugin to use
+    # +match+:: Parameters for the plugin
+    def self.modify(description, action, match={})
+      puts "============================="
+      puts  "MODIFY/CUSTOM: #{description}, ACTION NAME: #{action.to_s}"
+      if @filter_type == :request_mod
+        puts "TYPE: REQUEST"
+      elsif @filter_type == :response_mod
+        puts  "TYPE: RESPONSE"
+      else
+        raise ConfigurationLoadError, "used param 'block' without a filter type"
+      end
+      # Check if action name exists and create it
+      found_name = false
+      @valid_plugins.each { |plugin|
+        if plugin[:name].to_s == action.to_s
+          found_name = true
+          # Check if given plugin supports filter type
+          unless plugin[:class]::MODES.include? @filter_type
+            raise ConfigurationLoadError, 'Plugin ' + action.to_s + ' does not support filter type'
+          end
+          new_plugin = {:name => action,
+                        :object => plugin[:class].new(@filter_type, match[match.keys.first]),
+                        :description => description}
+          if @filter_type == :request_mod
+            @temp_req_mod << new_plugin
+          elsif @filter_type == :response_mod
+            @temp_resp_mod << new_plugin
+          end
+        end
+      }
+      unless found_name
+        raise ConfigurationLoadError, 'Unknown plugin type'
+      end
+    end
+    # Custom action, alias for modify
+    # The reason for it being a simple alias is that it may not modify
+    # its content, for example a virustotal scan takes too long
+    # so we pass it through anyway, but the scan results is saved in a log file
+    # May modify the content, but default behaviour should be that it doesn't
+    # +description+:: Description of the function
+    # +action+:: Name of the action in the configuration file
+    # +match+:: Attributes
+    def self.custom_action(description, action, match={})
+      self.modify(description, action, match)
+    end
+
+    # Placeholder if user wants to write pass, doesn't do anything
+    # For future expansion: Add conditionals to the rules ==> rewrite pass
+    def self.pass()
+    end
+    # Delete current filters
+    # Simple reset for tests of configuration files
+    def self.delete_filters()
+      @req_mod = nil
+      @resp_mod = nil
+      @temp_req_mod = nil
+      @temp_resp_mod = nil
+      @filter_type = nil
+    end
+
+    # Try to load the configureation file from path.
+    # If everything works as expected, the new filter rules are set,
+    # otherwise an error is raised if the configuration file is invalid and no previous configuration is available
+    # which would lead into an empty config or the loading will fail with an error on the log.
+    # +path+:: Path to the configuration file
+    def self.load_filters(path)
+      # puts are required because no logger is given when the service is created
+      # Setup objects to be written into
+      puts 'Trying to read new configuration'
+      # Set valid plugins and filters, has to done for each reload
+      @valid_filters = ICAPrb::Filters.get_filters
+      @valid_plugins = ICAPrb::Filters.get_plugins
+      # Execute file (we trust the user blindly, assuming that the configuration file is protected sufficiently)
+      @temp_resp_mod = []
+      @temp_req_mod = []
+      begin
+        eval(File.read File.expand_path(path))
+      rescue StandardError => error
+        error_info = "#{error.message} #{error.backtrace}"
+        # Initial info couldn't be loaded, shutting down filter framework
+        if @resp_mod == nil or @req_mod == nil
+          puts "Initial configuration couldn't be loaded: #{error_info}"
+          # Non recoverable
+          raise ConfigurationLoadError, "Initial configuration couldn't be loaded: #{error_info}"
+        else
+          puts "Error when reloading the configuration, configuration wasn't changed: #{error_info}"
+          # Don't throw an error, we still retain the old configuration
+          @temp_resp_mod = nil
+          @temp_req_mod = nil
+          @filter_type = nil
+          return
+        end
+      end
+      # Write data into variables (writing it beforehand is dangerous because the incomplete object may be used
+      # for new requests, this allows to reload the configuration during runtime)
+      @req_mod = @temp_req_mod
+      @resp_mod = @temp_resp_mod
+      puts 'Finished reading filter configuration'
+      return @req_mod, @resp_mod
+    end
+
+    # Set logger for filter
+    # +logger+:: Logger object
+    def self.set_logger(logger)
+      @logger = logger
+    end
+  end
+end