gnip_api 0.0.1 → 0.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: dbd9204b40905addd74370eaac461f346a222cdf
-  data.tar.gz: 86b9866f63329adf2e63f82431cb8dec3bf489cf
+  metadata.gz: 74edd623bf1f6bb85470235633add59192a90448
+  data.tar.gz: 51786aa16a0f9df12bf3e44595441096e0e0db1c
 SHA512:
-  metadata.gz: 3b0a11c8baa694d2e5da35342f7aaa1140386d3fd1f6da1fc9c9d90f4a67546d9d03400b72548315705a8f26e8a4d1889ae60ca7b7fee88ac82741ff5b9c4f56
-  data.tar.gz: de879b08e0fe1c5c7b0f6eed2dff14190c6418fa757b36933c89adee5d8b928791c123458489e6fb21e4aae32a0c594985dbaa2e6dd67ae35e18a3e82b3ddb24
+  metadata.gz: e49dafce62bb9f34c05127676b6099ee687c35667d849d609e241cf198c11196069b41f7ab979ee3a67a5f56d670bf4fd07cf7a1a5f1642aa367ff56724427f2
+  data.tar.gz: c88927e42d3da21e5b8ebd59ae391733d8f79a50e5d42c77c21498368a161827790329da94a80a100b66856a4f91b5c302407235d0c9ba72edce2434ee489aff

data/.gitignore

@@ -19,4 +19,5 @@ coverage/*
 *.iml
 .rvmrc
 dump.rdb
-test_env.rb
+test_env.rb
+pkg

data/.ruby-gemset

@@ -0,0 +1 @@
+gnip_api

data/.ruby-version

@@ -0,0 +1 @@
+ruby-2.1.0

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    gnip_api (0.0.1)
+    gnip_api (0.0.2)
       activesupport
       httparty
       json
@@ -45,6 +45,7 @@ GEM
     rspec-support (3.0.3)
     slop (3.6.0)
     thread_safe (0.3.5)
+    timecop (0.7.1)
     tzinfo (1.2.2)
       thread_safe (~> 0.1)
 
@@ -58,3 +59,7 @@ DEPENDENCIES
   pry
   rake (~> 10.0)
   rspec
+  timecop
+
+BUNDLED WITH
+   1.10.4

data/README.md

@@ -1,8 +1,13 @@
 # GnipApi
 
-You can connect with Gnip APIs with this gem and operate data. For now there's only a basic set of apis implemented, but I plan to include more in the future.
-The PowerTrack stream consumer is available as well as the listing of rules. Full rules operation will be included later.
-Also the gem will support different adapters, currently it's dependent of HTTParty, but I plan to remove this dependency and let the user choose which adapter to use, which can be a custom one of course. You will be able to define your own adapter to handle the http connections. 
+Connect with different Gnip APIs and get data from streams.
+
+## Notes
+
+- Can't access Search API currently
+- Rules endpoint are now rate limited.
+- A simple rate limiter was implemented.
+- You can configure the gem's mutex outside the gem if you plan to have concurrency. Very experimental for now.
 
 ## Installation
 
@@ -22,7 +27,39 @@ Or install it yourself as:
 
 ## Usage
 
-TODO
+### Configure the gem
+
+```ruby
+GnipApi.configure |config|
+  config.user = 'someone' # Gnip Account Username
+  config.password = 'something' # Gnip Password
+  config.account = 'myGnipAccount' # Your accounts name
+  config.adapter_class = SomeAdapter # You can define your own adapter, more in the following section
+  config.logger = Logger.new('myLog.log') # You can also provide a custom logger
+  config.mutex = Mutex.new # Experimental thread safety, more below
+end
+```
+
+Put the avobe code in a initializer if you're using Rails, or somewhere else if you aren't. After that you can interact with Gnip APIs.
+
+Note that you'll need a source and a label. Source is the data source within Gnip, such as Twitter, and label is the identifier of your stream.
+
+### Adapters
+
+GnipApi is not dependent of a single adapter (there's a dependency with HTTParty, but shhh... it won't last too long). You can use one of the provided adapters, or you can make your own, using the BaseAdapter class. You only need to implement the desired connector POST, GET and DELETE methods. There's an extra stream_get method, but it's just a variant of GET.
+
+The custom adapter does not require to live within the gem files, as long as GnipApi has access to your class, just put it in the config and you're ready to go. See the current adapters for reference.
+
+### Mutex and Thread Safety
+
+GnipApi has this feature in experimental stage, using a simple mutex to interact with the RateLimiter. The limiters are very simple, and should prevent your code executing more requests than it should. Depending where you are defining the mutex, you can elevate the coverage. GnipApi only requires an instance of Mutex, which you can place somewhere else for it to use.
+
+## WIP State
+
+GnipApi is still a WIP. Call it a beta, alpha, gem that has part of the features, whatever. Thing is, that it is currently usable. The custom adapter feature is there, and some of the APIs of Gnip were implemented. I'll be coding more things into this, such as other APIs, more limiters, request retries, error handling, different adapters for known connectors like RestClient or HTTP/net, etc.
+
+In any case, you were warned about it. Feel free to fill my issues list on Github :)
+
 
 ## Contributing
 
@@ -31,3 +68,5 @@ TODO
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create a new Pull Request
+
+This library was constructed with the help of [Armando Andini](https://github.com/antico5) who provided the basis to connect with the Gnip APIs.

data/gnip_api.gemspec

@@ -23,6 +23,7 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "pry"
   spec.add_development_dependency "awesome_print"       
   spec.add_development_dependency "rspec"
+  spec.add_development_dependency "timecop"
 
   spec.add_dependency "httparty"
   spec.add_dependency "json"

data/lib/gnip_api/adapter.rb

@@ -1,40 +1,62 @@
 module GnipApi
   class Adapter
+    GET = 'GET'
+    POST = 'POST'
+    DELETE = 'DELETE'
+
     attr_reader :adapter
 
     def initialize
-      @adapter = DevAdapter.new # GnipApi.configuration.adapter_class.new
+      raise GnipApi::Errors::MissingCredentials unless GnipApi.credentials?
+      raise GnipApi::Errors::MissingAdapter unless GnipApi.adapter_class?
+      @adapter = GnipApi.config.adapter_class.new
+      @logger = GnipApi.config.logger
     end
 
-    def get url
-      DevAdapter.get url
+    def get request
+      log_request(request)
+      response = adapter.get(request)
+      check_response_for_errors(response)
+      return response.body unless response.body.empty?
+      return true
     end
 
-    def stream_get url
-      DevAdapter.stream_get url do |data|
-        yield(data)
-      end
+    def post request
+      log_request(request)
+      response = adapter.post(request)
+      check_response_for_errors(response)
+      return response.body unless response.body.empty?
+      return true
     end
-  end
 
-  class DevAdapter
-    class << self
-      def stream_get url
-        HTTParty.get url, :basic_auth => auth do |data|
-          yield(data)
-        end
-      end
+    def delete request
+      log_request(request)
+      response = adapter.delete(request)
+      check_response_for_errors(response)
+      return response.body unless response.body.empty?
+      return true
+    end
 
-      def get url
-        HTTParty.get url, :basic_auth => auth
+    def stream_get request
+      log_request(request)
+      adapter.stream_get(request) do |data|
+        yield(data)
       end
+    end
 
-      def auth
-        {
-          :username => GnipApi.configuration.user, 
-          :password => GnipApi.configuration.password
-        }
+    def check_response_for_errors response
+      if response.ok?
+        @logger.info "#{response.request_method} request to #{response.request_uri} returned with status #{response.status} OK"
+      else
+        error_message = response.error_message
+        @logger.error "#{response.request_method} request to #{response.request_uri} returned with status #{response.status} FAIL: #{error_message}"
+        raise GnipApi::Errors::Adapter::RequestError.new, error_message
       end
     end
+
+    private
+    def log_request request
+      @logger.info "Starting #{request.request_method} request to #{request.uri}"
+    end
   end
 end

data/lib/gnip_api/adapters/base_adapter.rb

@@ -0,0 +1,59 @@
+# BaseAdapter should be used on any adapter for this gem. In order
+# to create your own adapter you must inherit this class to 
+# let your adapter use the provided functionality of the gem. In 
+# other words, this just defines basic methods to provide the 
+# custom adapter with basic data.
+#
+# To create a custom adapter, create your class, and implement the
+# following methods that GnipApi::Adapter will need.
+#
+# - #get
+# - #post
+# - #delete
+# - #stream_get
+#
+# All single-request methods (stream_get not included), should
+# return a GnipApi::Response object with:
+#
+# - status: integer value for status code (200, 403, etc)
+# - body: raw body, no matter if it's json or xml, the implementation of the api will know this
+# - headers: a hash with the response headers
+#
+# The response object is an effort to standarize the data and let Gnip::Adapter 
+# operate without the concern of the data format. The helper method to create
+# this response is available in this class, and will be available to your class
+# when you write your adapter.
+#
+# The process of parsing the headers to a hash, setting authorization headers or
+# parsing response data to break it down to status, body and headers are responsabilities
+# of your custom adapter class.
+#
+# Gnip::Adapter will pass your adapter a Gnip::Request object that contains:
+#
+# - uri: URI object with url, schema, etc
+# - payload: a parsed text to send in the request's body
+# - headers: a hash that may contain additional headers
+#
+# Your adapter should process this object and translate it into a relevant request for
+# your desired connector. Note that payload and headers may be nil if not needed.
+#
+# Last thing to denote, GnipApi::Adapter will expect an instance of your adapter to use.
+# What you do inside your adapter or how you implement a specific adapter is up to you.
+
+module GnipApi
+  module Adapters
+    class BaseAdapter
+      def username
+        GnipApi.configuration.user
+      end
+      
+      def password
+        GnipApi.configuration.password
+      end
+      
+      def create_response request, status, body, headers
+        GnipApi::Response.new request, status, body, headers
+      end
+    end
+  end
+end

data/lib/gnip_api/adapters/httparty_adapter.rb

@@ -1,6 +1,38 @@
 module GnipApi
   module Adapters
-    class HTTPartyAdapter
+    class HTTPartyAdapter < GnipApi::Adapters::BaseAdapter
+      def post request
+        data = HTTParty.post request.uri, :basic_auth => auth, :body => request.payload
+        return response(request, data)
+      end
+      
+      def delete request
+        data = HTTParty.delete request.uri, :basic_auth => auth, :body => request.payload
+        return response(request, data)
+      end
+      
+      def get request
+        data = HTTParty.get request.uri, :basic_auth => auth
+        return response(request, data)
+      end
+
+      def stream_get request
+        HTTParty.get request.uri, :headers => request.headers, :basic_auth => auth do |data|
+          yield(data)
+        end
+      end
+
+      def auth
+        {
+          :username => username,
+          :password => password
+        }
+      end
+      
+      def response request, data
+        create_response request, data.code, data.body, data.headers
+      end
+      
     end
   end
 end

data/lib/gnip_api/apis/power_track/buffer.rb

@@ -9,6 +9,10 @@ module GnipApi
           @data = ""
         end
         
+        def size
+          @data.size
+        end
+        
         def insert! chunk
           @data << chunk
         end

data/lib/gnip_api/apis/power_track/rule.rb

@@ -0,0 +1,31 @@
+module GnipApi
+  module Apis
+    module PowerTrack
+      class Rule
+        attr_accessor :value, :tag
+
+        def initialize params={}
+          @value = params[:value] || params['value']
+          @tag = params[:tag] || params['tag']
+        end
+
+        def to_json
+          attributes.to_json
+        end
+
+        def attributes
+          attrs = {}
+          attrs[:value] = @value if @value
+          attrs[:tag] = @tag if @tag
+          attrs
+        end
+
+        def uid
+          rule = @value
+          rule += "tag:#{@tag}" if @tag
+          Digest::SHA2.hexdigest(rule)
+        end
+      end
+    end
+  end
+end

data/lib/gnip_api/apis/power_track/rules.rb

@@ -1,15 +1,82 @@
+# Gnip PowerTrack Rules API
+#
+# Create, delete and list rules of a powetrack stream.
+
 module GnipApi
   module Apis
     module PowerTrack
       class Rules
         attr_reader :adapter
 
-        def initialize
+        # In order to do any operation, you need to specify:
+        # - label: the label of your stream
+        # - source: which data source to use (I think only twitter is available)
+        def initialize params={}
           @adapter = GnipApi::Adapter.new
+          @label = params[:label]
+          @source = params[:source]
         end
 
+        # Returns an array of defined rules
         def list
-          adapter.get GnipApi::Endpoints.powertrack_rules('twitter', 'prod')
+          request_allowed?
+          request = create_get_request
+          rules = adapter.get(request)
+          parse_rules(rules)
+        end
+
+        # Creates the specified rule. Parameters:
+        # - rules: GnipApi::Apis::PowerTrack::Rule object
+        def create rules
+          raise GnipApi::Errors::PowerTrack::MissingRules.new if rules.nil? || rules.empty?
+          request_allowed?
+          request = create_post_request(construct_rules(rules))
+          adapter.post(request)
+        end
+
+        # Deletes the specified rule. Parameters:
+        # - rules: GnipApi::Apis::PowerTrack::Rule object
+        def delete rules
+          raise GnipApi::Errors::PowerTrack::MissingRules.new if rules.nil? || rules.empty?
+          request_allowed?
+          request = create_delete_request(construct_rules(rules))
+          adapter.delete(request)
+        end
+
+        # Parses an array of GnipApi::Apis::PowerTrack::Rule objects
+        # to the necesary JSON format for the endpoint
+        def construct_rules rules
+          parsed_rules = {:rules => []}
+          rules.each do |rule|
+            parsed_rules[:rules] << rule.attributes
+          end
+          parsed_rules.to_json
+        end
+
+        def parse_rules data
+          parsed_data = JSON.parse(data)
+          parsed_data['rules'].map{|rule| GnipApi::Apis::PowerTrack::Rule.new(:value => rule['value'], :tag => rule['tag'])}
+        end
+
+        private
+        def endpoint
+          GnipApi::Endpoints.powertrack_rules(@source, @label)
+        end
+
+        def create_get_request
+          GnipApi::Request.new_get(endpoint)
+        end
+
+        def create_post_request payload
+          GnipApi::Request.new_post(endpoint, payload)
+        end
+        
+        def create_delete_request payload
+          GnipApi::Request.new_delete(endpoint, payload)
+        end
+
+        def request_allowed?
+          raise 'RateLimited' unless GnipApi.config.rate_limiter.rules_request_allowed?
         end
       end
     end

data/lib/gnip_api/apis/power_track/stream.rb

@@ -2,7 +2,7 @@ module GnipApi
   module Apis
     module PowerTrack
       class Stream
-        attr_reader :system_messages, :adapter
+        attr_reader :adapter
         
         def initialize params = {}
           @stream = params[:stream]
@@ -15,15 +15,53 @@ module GnipApi
         end
 
         def consume
-          adapter.stream_get endpoint do |chunk|
+          request = create_request
+          adapter.stream_get request do |chunk|
             @buffer.insert! chunk
-             yield(@buffer.read!)
+            begin
+              yield process_entries(@buffer.read!)
+            rescue Exception => e
+              puts e.class
+              puts e.message
+              puts e.backtrace[0..10].join("\n")
+              raise e
+            end
+          end
+        end 
+
+        def process_entries entries
+          entries.map!{|e| parse_json(e)}.compact!
+          entries.map!{|e| build_message(e)}
+          log_system_messages(entries)
+          entries
+        end
+
+        def log_system_messages entries
+          entries.select{|message| message.system_message? }.each do |system_message|
+            GnipApi.logger.send(system_message.log_method, system_message.message)
+          end
+        end
+
+        def build_message params
+          Gnip::Message.build(params)
+        end
+
+        def parse_json json
+          begin 
+            JSON.parse json
+          rescue JSON::ParserError
+            nil
           end
         end
 
         private
+        def create_request 
+          GnipApi::Request.new_get(endpoint, {'Accept-Encoding' => 'gzip'})
+        end
 
         def set_config
+          raise 'MissingStream' if @stream.nil?
+          raise 'MissingSource' if @source.nil?
           @user = GnipApi.configuration.user
           @password = GnipApi.configuration.password
           @account = GnipApi.configuration.account
@@ -31,29 +69,10 @@ module GnipApi
           @buffer = GnipApi::Apis::PowerTrack::Buffer.new
         end
 
-        def handle message
-          @system_messages << message if message.system_message?
-          yield message if message.activity?
-        end
-
         def endpoint
           GnipApi::Endpoints.powertrack_stream(@source, @stream)
         end
 
-        def parse data
-          json_parse(extract_messages(data)).map { |hash| Message.build hash }
-        end
-
-        def json_parse messages
-          messages.map do |message|
-            begin 
-              JSON.parse message
-            rescue JSON::ParserError
-              nil
-            end
-          end.compact
-        end
-
       end
     end
   end

data/lib/gnip_api/apis/search/query.rb

@@ -0,0 +1,36 @@
+module GnipApi
+  module Apis
+    module Search
+      class Query
+        attr_accessor :rules, :from_date, :to_date, :max_results, :next, :publisher
+        
+        def initialize params={}
+          @rules = params[:rules]
+          @from_date = params[:from_date]
+          @to_date = params[:to_date]
+          @max_results = params[:max_results]
+          @next = params[:next]
+          @publisher = params[:publisher]
+        end
+
+        def to_json
+          hash = {}
+          hash['query'] = @rules
+          hash['publisher'] = @publisher
+          hash['fromDate'] = @from_date.to_time.utc.strftime('%Y%m%d%H%M') if @from_date
+          hash['toDate'] = @to_date.to_time.utc.strftime('%Y%m%d%H%M') if @to_date
+          hash['next'] = @next if @next
+          hash['maxResults'] = @max_results if @max_results
+          hash.to_json
+        end
+
+        def valid?
+        end
+
+        private
+        def validate
+        end
+      end
+    end
+  end
+end

data/lib/gnip_api/apis/search/result.rb

@@ -0,0 +1,17 @@
+module GnipApi
+  module Apis
+    module Search
+      class Result
+        attr_reader :activities, :response
+        
+        def initialize response
+          @response = response
+        end
+        
+        def activities
+          @activities ||= response['results'].map { |activity| Activity.new(activity) }
+        end
+      end
+    end
+  end
+end

data/lib/gnip_api/apis/search/stream.rb

@@ -0,0 +1,26 @@
+module GnipApi
+  module Apis
+    module Search
+      class Stream
+        attr_accessor :query, :label
+
+        def initialize options={}
+          @query = options[:query]
+          @label = options[:label]
+          @adapter = GnipApi::Adapter.new
+        end
+
+        def perform
+          response = @adapter.post search_url, @query.to_json
+          Result.new response
+        end
+
+        private
+
+        def search_url
+          GnipApi::Endpoints.search_stream(@label)
+        end
+      end
+    end
+  end
+end

data/lib/gnip_api/configuration.rb

@@ -1,10 +1,13 @@
 module GnipApi
   class Configuration
-    attr_accessor :user, :password, :adapter_class, :account, :logger
+    attr_accessor :user, :password, :adapter_class, :account, :logger, :mutex
+    attr_reader :rate_limiter
 
     def initialize
       @adapter_class = GnipApi::Adapters::HTTPartyAdapter
       @logger = Logger.new('tmp/gnip_api.log')
+      @mutex = Mutex.new
+      @rate_limiter = GnipApi::RateLimiter.new
     end
   end
 end

data/lib/gnip_api/endpoints.rb

@@ -2,11 +2,24 @@ module GnipApi
   class Endpoints
     class << self
       def powertrack_rules source, label
-        URI("https://api.gnip.com:443/accounts/#{GnipApi.configuration.account}/publishers/#{source}/streams/track/#{label}/rules.json")
+        URI("https://api.gnip.com:443/accounts/#{account}/publishers/#{source}/streams/track/#{label}/rules.json")
       end
 
       def powertrack_stream source, label
-        URI("https://stream.gnip.com:443/accounts/#{GnipApi.configuration.account}/publishers/#{source}/streams/track/#{label}.json")
+        URI("https://stream.gnip.com:443/accounts/#{account}/publishers/#{source}/streams/track/#{label}.json")
+      end
+
+      def search_stream label
+        URI("https://search.gnip.com/accounts/#{account}/search/#{label}.json")
+      end
+      
+      def search_count
+        URI("https://search.gnip.com/accounts/#{ CONFIG['account'] }/search/#{label}.json/counts")
+      end
+
+      private
+      def account
+        GnipApi.configuration.account
       end
     end
   end