google-safe-browsing-plugin 0.1.1

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2013 stonelonely
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,88 @@
+## Google Safe Browsing Plugin
+
+A Rails 3 plugin for [Google Safe Browsing API v2](https://developers.google.com/safe-browsing/developers_guide_v2).
+It supports Google malware and phishing list.
+
+## Installation
+
+Add the plugin to your Gemfile
+
+    gem 'google_safe_browsing_plugin', '~> 0.1'
+
+After bundle install, run the following to generate the db migration, model classes and other code
+
+	bundle install
+    bundle exec rails g google:install
+
+Run the migrations generated from the previous step, and then seed the databse
+
+    bundle exec rake db:migrate
+	bundle exec rake google:safe_browsing:db_seed
+	
+Edit the configration file with your Google API key
+    
+	# Edit config/google_safe_browsing.yml, and replace the real API key in line 2.
+	
+
+## Build the hash prefix data locally
+
+The plugin stores the hash prefixes in the relational database. The following rake task needs to be run under a _cron schedule_ to keep the local data in sync with the Google server lists. It may require several runs initially before you have a relatively complete hash prefix set before you can do any meaningful full hash lookup. The first run may take a while because it needs to download quite a bit of data and store them in the local database. The initial run of the rake task could generate quarter a million shavar records in the database.
+
+    bundle exec rake google:safe_browsing:load_remote
+
+## Url lookup
+
+After you run the _'load_remote'_ rake task several times, your local cache of the hash prefixes will be ready. Now you can start to do url lookup. Start the Rails console, and then try the following
+
+    url = 'financestudyhelp.com'
+	r = Google::SafeBrowsingHelper.lookup_url url
+	
+Since the Google Safe Browsing data get updated frequently, the previous query may not necessarily generate hit on Malware, other urls you can try are 'http://gumblar.cn' and 'http://ianfette.org'.
+
+Upon a match on full-length hash lookup, the _lookup\_url_ call will return a hash object that contains the match. E.g. 
+
+    {"financestudyhelp.com"=>["goog-malware-shavar"]}
+
+The key of the hash is the url that's been queried. The array is the match themselves. If the url is both a malware and 
+a phishing link, the value will be 
+    
+	["goog-malware-shavar","googpub-phish-shavar"]
+	
+If the url is neither a malware nor phishing link, the lookup result will be an empty array [].
+
+
+## Uninstall
+
+If you want to uninstall the gem and remove the generated files
+   
+	rails d google:install
+
+## Features and limitations
+
+* The plugin does not hide the ActiveRecord models and the helper inside the Gem. It instead uses the generator function provided by Rails and Thor to copy the template model/migration/helper code to the Rails application source tree. The generated code is under the _Google_ namespace.
+* The plugin uses the relational database as the data store. But it doesn't have to. Redis may be a better choice considering lookup speed and cache expire. But the local data have to be kept under certain limit. The tables generated from the migration have _google_ as the prefix.
+* The hash prefix is stored in the database as plain text but not in binary/encoded format.
+* The config parameters, in _config/google\_safe\_browsing.yml_, can be changed to your need. E.g. _full\_length\_hash\_expires_ is for how long the full-length hash will be cached locally.
+* The plugin has backoff strategy built in when error happens. So if you notice that the full-length hash request doesn't go to Google, it probably is still in the backoff mode that prevents the request being sent to Google.
+* Some RSpec tests are provided in the spec folder.
+* The _google:safe\_browsing:load\_remote_ rake task cannot be run repeatedly within a short time span. It honors the _NEXT_ instruction from Google's response. This value is kept in _google\_functions#next\_updated\_at_. You have to change the value if you want to run it immediately after the previous run.
+* When Google sends out _'reset'_ instruction to the request, the plugin will _not_ clean up the local data by default. You can change the behavior to reset the data by adding the following configuration in application.rb or the environment file in your Rails app:
+<pre><code>
+	# This will reset your local data
+    config.google_safe_browsing_upon_reset = lambda {
+      Google::SafeBrowsingShavar.delete_all
+    }
+</code></pre>
+
+* _'rekey'_ instruction is not supported.
+* The plugin works for Rails 3.2. But it should be relatively trivial to make it work with other Rails version. The reason for using Rails is mostly because of the value ActiveRecord provides for the data mapping and store. The download, parsing and encoding/decoding of the Google Safe Browsing data do not have to use Rails.
+
+
+## Reference
+
+* [Safe Browsing API v2](https://developers.google.com/safe-browsing/developers_guide_v2)
+* [google-safe-browsing](http://code.google.com/p/google-safe-browsing/wiki/Protocolv2Spec) project on Google code.
+
+
+
+Copyright (c) 2013 stonelonely and contributors, released under the MIT license.

data/lib/faraday/response/safe_browsing_update_parser.rb

@@ -0,0 +1,119 @@
+require 'faraday'
+
+module Faraday
+  class Response
+    
+    class SafeBrowsingUpdateParser < ::Faraday::Response::Middleware
+
+      REKEY ||= /(e):(pleaserekey)/
+      NEXT  ||= /(n):(\d+)/
+      RESET ||= /(r):(pleasereset)/
+      LIST  ||= /(i):(.+)/
+      
+      MIX_LINE ||= /(i|u|ad|sd):(.+)/
+      CHUNK_LIST ||= /(\d+-\d+|\d+)/
+      
+      # define_parser do |body|
+      def parse body
+        @update_obj = Google::SafeBrowsingUpdate.new
+        parse_data_response(body)
+        @update_obj
+      end
+      
+      # 
+      # BODY      = [(REKEY | MAC) LF] NEXT LF (RESET | (LIST LF)+) EOF
+      # NEXT      = "n:" DIGIT+                               # Minimum delay before polling again in seconds
+      # REKEY     = "e:pleaserekey"
+      # RESET     = "r:pleasereset"
+      # LIST      = "i:" LISTNAME [MAC] (LF LISTDATA)+
+      # LISTNAME  = (LOALPHA | DIGIT | "-")+                  # e.g. "goog-phish-sha128"
+      # MAC       = "," (LOALPHA | DIGIT)+
+      # LISTDATA  = ((REDIRECT_URL | ADDDEL-HEAD | SUBDEL-HEAD) LF)+
+      # REDIRECT_URL = "u:" URL [MAC]
+      # URL       = Defined in RFC 1738
+      # ADDDEL-HEAD  = "ad:" CHUNKLIST
+      # SUBDEL-HEAD  = "sd:" CHUNKLIST
+      # CHUNKLIST = (RANGE | NUMBER) ["," CHUNKLIST]
+      # NUMBER    = DIGIT+                                    # Chunk number >= 1
+      # RANGE     = NUMBER "-" NUMBER
+      # 
+      def parse_data_response body
+        lines = body.split(%Q(\n))
+        text = lines.shift
+        if REKEY =~ text
+          @update_obj.rekey = true
+          text = lines.shift
+        end
+        # the line is NEXT line
+        parse_next text
+        return if @update_obj.rekey
+        
+        text = lines.shift
+        if RESET =~ text
+          @update_obj.reset = true
+          return
+        end
+        # the line is the LIST line
+        parse_list text
+        while !(text = lines.shift).nil?
+          parse_mix_line text
+        end
+      end
+      
+      
+      def parse_next line
+        m = NEXT.match(line.to_s)
+        if m.nil?
+          raise Google::Error::ParserError
+        else
+          @update_obj.next = m[2].to_i
+        end
+      end
+      
+      def parse_list line
+        m = LIST.match(line.to_s)
+        if m.nil?
+          raise Google::Error::ParserError
+        else
+          @update_obj.set_current_list(m[2])
+        end
+      end
+    
+      def parse_mix_line line
+        m = MIX_LINE.match(line.to_s)
+        raise Google::Error::ParserError if m.nil?
+        my_list = @update_obj.get_current_list
+        
+        case m[1]
+        when 'i'
+          @update_obj.set_current_list m[2]
+        when 'u'
+          my_list[:u] << m[2].strip
+        when 'sd'
+          parse_chunk_list m[2], my_list[:sd]
+        when 'ad'
+          parse_chunk_list m[2], my_list[:ad]
+        else
+          raise Google::Error::ParserError
+        end
+      end
+      
+      def parse_chunk_list chunk_list, cached_chunks
+        chunks = chunk_list.split(',')
+        while !(chunk = chunks.shift).nil?
+          m = CHUNK_LIST.match(chunk)
+          raise Google::Error::ParserError if m.nil?
+          if m[0].include?('-')
+            low, upper = m[0].split('-').map {|x| x.to_i}
+            cached_chunks << (low..upper)  # Range
+          else
+            cached_chunks << m[0].to_i
+          end
+        end
+      end
+
+    end
+  
+  end
+end
+    

data/lib/google/safe_browsing_client.rb

@@ -0,0 +1,211 @@
+require 'faraday'
+
+module Google
+  class SafeBrowsingClient
+    
+    include Google::SafeBrowsingUpdateHelper
+    
+    attr_accessor :headers
+  
+    CLIENT  ||= 'api'
+    API_KEY ||= Google::CONFIG['api_key']
+    APP_VER ||= Google::CONFIG['app_ver']
+    P_VER   ||= Google::CONFIG['p_ver']
+  
+    SEMI_COLON ||= ';'
+    COLON      ||= ':'
+    NL ||= %Q(\n)
+    
+    FULL_HASH_TIMEOUT = 2 # secs
+    
+    def initialize
+      @headers = {
+        :'User-Agent'    => 'Faraday Ruby Client'
+      }
+    end
+
+    def api_server
+      'https://safebrowsing.clients.google.com'
+    end
+  
+    #
+    # goog-malware-shavar
+    # goog-regtest-shavar
+    # goog-whitedomain-shavar
+    # googpub-phish-shavar
+    #
+    def list_url
+      '/safebrowsing/list?client=%s&apikey=%s&appver=%s&pver=%s' % [CLIENT, API_KEY, APP_VER, P_VER]
+    end
+    
+    def download_url
+      '/safebrowsing/downloads?client=%s&apikey=%s&appver=%s&pver=%s' % [CLIENT, API_KEY, APP_VER, P_VER]
+    end  
+
+    def full_hash_url
+      '/safebrowsing/gethash?client=%s&apikey=%s&appver=%s&pver=%s' % [CLIENT, API_KEY, APP_VER, P_VER]
+    end
+
+    #
+    #       resp = <<-END_OF_SAMPLE
+    # n:1200
+    # i:googpub-phish-shavar
+    # u:cache.google.com/first_redirect_example
+    # u:cache.google.com/second_redirect_example
+    # sd:1,2
+    # i:acme-white-shavar
+    # u:cache.google.com/second_redirect_example
+    # ad:1-2,4-5,7
+    # sd:2-6
+    #       END_OF_SAMPLE
+    #       
+    #       parser = Faraday::Response::SafeBrowsingUpdateParser.new
+    # r = parser.parse resp
+    # 
+    def shavar_data_update list_name, options = {}
+      conn = Faraday.new(url: api_server, headers: headers) do |builder|
+        builder.use Faraday::Request::UrlEncoded
+        builder.use Faraday::Response::SafeBrowsingUpdateParser
+        builder.adapter ::Faraday.default_adapter
+      end
+      
+      request_body = download_data_request_body(list_name)
+      
+      r = conn.post do |req|
+        req.url download_url
+        req.body = request_body
+      end
+
+      if successful_status_code?(r.status)
+        r.body
+      else
+        process_error_response(r.status)
+      end
+    end
+    
+    # 
+    # To get shavar chunk data from the redirect url
+    #
+    def chunk_data redirect_url, options = {}
+      url = 'https://' + redirect_url if /\Ahttps?/ !~ redirect_url
+      
+      conn = Faraday.new(headers: headers) do |builder|
+        builder.use Faraday::Request::UrlEncoded
+        builder.adapter ::Faraday.default_adapter
+      end
+
+      r = conn.post do |req|
+        req.url url
+      end
+
+      if successful_status_code?(r.status)
+        r.body
+      else
+        process_error_response(r.status)
+      end
+
+    end
+    
+    # 
+    # hash_prefixes: ['5b3583c0', 'b3e357a6']
+    # @return 
+    #     # { 'goog-malware-shavar' (list_name) 
+    #      => {
+    #          :add_chunk_num1 => [full_hash0, full_hash1, ...]
+    #          :add_chunk_num2 => [full_hash0]
+    #         }
+    # }
+    #
+    def full_hash hash_prefixes
+      conn = Faraday.new(url: api_server, headers: headers) do |builder|
+        builder.use Faraday::Request::UrlEncoded
+        builder.adapter ::Faraday.default_adapter
+      end
+      
+      request_body = full_hash_request_body hash_prefixes
+      
+      r = conn.post do |req|
+        req.url full_hash_url
+        req.body = request_body 
+        req.options[:timeout] = FULL_HASH_TIMEOUT
+      end
+            
+      if successful_status_code?(r.status)
+        SafeBrowsingParser.parse_full_hash_entries r.body
+      else
+        process_error_response(r.status)
+      end
+    end
+    
+    # 
+    # s;200
+    # googpub-phish-shavar;a:1-3,5,8:s:4-5
+    # acme-white-shavar;a:1-7:s:1-2
+    # 
+    def download_data_request_body list_name
+      gen_list_request(list_name)
+    end
+    
+    def gen_list_request list
+      list = SafeBrowsingList.where(name: list).first
+      s = StringIO.new("")
+      s << list.name << SEMI_COLON
+      add_chunk_ids = gen_chunk_nums_string(
+                          SafeBrowsingShavar.add_chunk_nums_for_list(list.name).map(&:chunk_num))
+      sub_chunk_ids = gen_chunk_nums_string(
+                          SafeBrowsingShavar.sub_chunk_nums_for_list(list.name).map(&:chunk_num))
+      
+      s << SafeBrowsingShavar::CHUNK_TYPE_ADD << COLON << add_chunk_ids unless add_chunk_ids.blank?
+      s << COLON if !add_chunk_ids.blank? && !sub_chunk_ids.blank?
+      s << SafeBrowsingShavar::CHUNK_TYPE_SUB << COLON << sub_chunk_ids unless sub_chunk_ids.blank?
+      s << NL
+      s.string
+    end
+    
+    # 
+    # BODY       = HEADER LF PREFIXES EOF
+    # HEADER     = PREFIXSIZE ":" LENGTH
+    # PREFIXSIZE = DIGIT+         # Size of each prefix in bytes
+    # LENGTH     = DIGIT+         # Size of PREFIXES in bytes
+    # 
+    # The prefixes should all have the same length of bytes
+    #
+    def full_hash_request_body prefixes
+      return "" if prefixes.empty?
+      
+      prefix_size = prefixes.first.size / 2
+      
+      s = StringIO.new("")
+      s << prefix_size.to_s << ":"
+      s << (prefix_size * prefixes.size).to_s << NL
+      s << pack_hash_prefix(prefixes, prefix_size)
+      s.string
+    end
+    
+    def pack_hash_prefix prefixes, prefix_size
+      s = StringIO.new("")
+      prefixes.each do |pre|
+        s << [pre].pack("H#{prefix_size*2}")
+      end
+      s.string
+    end
+    
+    def process_error_response status
+      case status.to_s
+      when /\A204/
+        raise Error::NoContent
+      when /\A4/
+        raise Error::InvalidRequest
+      when /\A5/
+        raise Error::ServiceUnavailable
+      else
+        raise Error::UnknownError
+      end
+    end
+
+    def successful_status_code?(status_code)
+      status_code.to_s == '200'
+    end
+    
+  end
+end

data/lib/google/safe_browsing_parser.rb

@@ -0,0 +1,214 @@
+module Google
+  module SafeBrowsingParser
+    extend self
+    
+    ADD ||= SafeBrowsingShavar::CHUNK_TYPE_ADD
+    SUB ||= SafeBrowsingShavar::CHUNK_TYPE_SUB
+    
+    ADD_SUB_HEAD ||= /(?<add_sub>a|s):(?<chunk_num>\d+):(?<hash_len>\d+):(?<chunk_len>\d+)(\n)/
+    FULL_HASH_HEAD ||= /(?<rekey>e:pleaserekey)|((?<list>[-_\w]+):(?<chunk_num>\d+):(?<chunk_len>\d+))(\n)/
+    
+    CHUNKNUM_SIZE = HOST_KEY_SIZE = 4 # Bytes
+
+    FULL_HASH_SIZE = 32 # Bytes, 256 bit
+    
+    #
+    # @params str A clob of characters returned from the redirect download url
+    # @returns Two arrays of shavar list data decoded: one for ADD, the other for SUB
+    # 
+    # The shavar list data has the following structure
+    # For ADD
+    # { :chunk_num => 343243, :hash_len => 4, :chunk_len => 4343,
+    #   :chunk_data => {
+    #                    :host_key_one => [prefix0, prefix1, ...],
+    #                    :host_key_two => []
+    #                  }
+    # }
+    # 
+    # For SUB
+    # { :chunk_num => 343243, :hash_len => 4, :chunk_len => 4343,
+    #   :chunk_data => {
+    #                    :host_key_one => { 
+    #                                       :add_chunknum_one => [prefix0, prefix1, ...],
+    #                                       :add_chunknum_two => []
+    #                                     }
+    #                    :host_key_two => {
+    #                                       :add_chunknum_three => [prefix0, prefix1, ...],
+    #                                       :add_chunknum_four => []
+    #                                     }
+    #                  }
+    # }
+    #
+    def parse_shavar_list str, test_mode = false
+      adds = []; subs = []
+      scanner = StringScanner.new(str)
+            
+      count = 0; scanner.pos = 0
+      while !(head = scanner.scan_until(ADD_SUB_HEAD)).nil?
+        if test_mode && count > 0
+          break
+        end
+        
+        count += 1
+        m = ADD_SUB_HEAD.match head
+        chunk_num, hash_len, chunk_len = m[:chunk_num].to_i, m[:hash_len].to_i, m[:chunk_len].to_i
+        pointer = 0; chunk_data = []
+
+        while pointer < chunk_len
+          chunk_data << scanner.get_byte
+          pointer += 1
+        end
+
+        if m[:add_sub] == 'a'
+          data_arr = parse_add_data(chunk_data, hash_len)
+        elsif m[:add_sub] == 's'
+          data_arr = parse_sub_data(chunk_data, hash_len)
+        end
+          
+        obj = {
+            chunk_num: chunk_num,
+            hash_len:  hash_len,
+            chunk_len: chunk_len,
+            chunk_data: data_arr
+          }
+
+        if m[:add_sub] == ADD
+          adds << obj
+        elsif m[:add_sub] == SUB
+          subs << obj
+        end
+        
+      end
+      
+      Rails.logger.info "Total # of ADD/SUB section is #{count}, #{adds.size} adds, #{subs.size} subs"
+      
+      [adds, subs]
+    end
+    
+    
+    def parse_add_data byte_arr, hash_len
+      ret = {}
+      total_chars = 0
+      pointer = 0
+      while pointer < byte_arr.size
+        host_key = parse_host_key byte_arr[pointer...pointer+HOST_KEY_SIZE]
+        total_chars += HOST_KEY_SIZE
+        pointer += HOST_KEY_SIZE
+        ret[host_key] ||= []
+        count = parse_count_number byte_arr[pointer]
+        pointer += 1
+        total_chars += 1
+        
+        if count > 0
+          sub_count = 0
+          while sub_count < count
+            ret[host_key] << parse_hash_prefix(byte_arr[pointer...pointer+hash_len], hash_len)
+            total_chars += hash_len
+            pointer += hash_len
+            sub_count += 1
+          end
+        end
+      end
+      
+      ret
+    end
+    
+    def parse_sub_data byte_arr, hash_len
+      ret = {}
+      total_chars = 0
+      pointer = 0
+      while pointer < byte_arr.size
+        host_key = parse_host_key byte_arr[pointer...pointer+HOST_KEY_SIZE]
+        total_chars += HOST_KEY_SIZE
+        pointer += HOST_KEY_SIZE
+        count = parse_count_number byte_arr[pointer]
+        total_chars += 1
+        pointer += 1
+
+        ret[host_key] ||= {}
+        if count > 0
+          sub_count = 0
+          while sub_count < count
+            add_chunknum = byte_arr[pointer...pointer+CHUNKNUM_SIZE].join('').unpack('L>').first
+            pointer += CHUNKNUM_SIZE; total_chars += CHUNKNUM_SIZE
+            ret[host_key][add_chunknum] ||= []
+            ret[host_key][add_chunknum] << parse_hash_prefix(byte_arr[pointer...pointer+hash_len], hash_len)
+            pointer += hash_len; total_chars += hash_len
+            sub_count += 1
+          end
+        else
+          add_chunknum = byte_arr[pointer...pointer+CHUNKNUM_SIZE].join('').unpack('L>').first
+          ret[host_key][add_chunknum] = []
+          pointer += CHUNKNUM_SIZE; total_chars += CHUNKNUM_SIZE
+        end
+        
+      end
+      
+      ret
+    end
+    
+    def parse_host_key char_arr
+      char_arr.join('').unpack('H8').first
+    end
+    
+    def parse_count_number char
+      char.unpack('C').first
+    end
+    
+    def parse_hash_prefix char_arr, hash_len
+      char_arr.join('').unpack("H#{hash_len*2}").first
+    end
+    
+    #
+    # Returns
+    # { 'goog-malware-shavar' (list_name) 
+    #      => {
+    #          :add_chunk_num1 => [full_hash0, full_hash1, ...]
+    #          :add_chunk_num2 => [full_hash0]
+    #         }
+    # }
+    #
+    # BODY        = ([MAC LF] HASHENTRY+) | (REKEY LF) EOF
+    # HASHENTRY   = LISTNAME ":" ADDCHUNK ":" HASHDATALEN LF HASHDATA
+    # ADDCHUNK    = DIGIT+                          # Add chunk number
+    # HASHDATALEN = DIGIT+                          # Length of HASHDATA
+    # HASHDATA    = <HASHDATALEN number of unsigned bytes>  # Full length hashes in binary
+    # MAC         = (LOALPHA | DIGIT)+
+    # 
+    # Ignore rekey response for now
+    #
+    def parse_full_hash_entries str
+      full_list = {}
+      scanner = StringScanner.new(str)
+      count = 0; scanner.pos = 0
+
+      while !(head = scanner.scan_until(FULL_HASH_HEAD)).nil?
+        m = FULL_HASH_HEAD.match head
+        return full_list if m[:rekey]
+        
+        count += 1
+        list_name, chunk_num, chunk_len = m[:list].to_s.to_sym, m[:chunk_num].to_i, m[:chunk_len].to_i
+        pointer = 0; chunk_data = []
+        
+        my_list = (full_list[list_name] ||= {})
+        my_list[chunk_num] ||= []
+        
+        while pointer < chunk_len
+          chunk_data << scanner.get_byte
+          pointer += 1
+        end
+
+        parse_full_hash_data chunk_data, my_list[chunk_num]
+      end
+      
+      full_list
+    end
+    
+    def parse_full_hash_data byte_arr, full_hash_arr
+      byte_arr.each_slice(FULL_HASH_SIZE) do |slice|
+        full_hash_arr << slice.join('').unpack("H#{FULL_HASH_SIZE*2}").first
+      end
+    end
+
+  end
+end