suj-pusher 0.2.0 → 0.2.3

data/README.md

@@ -4,11 +4,11 @@ This is a simple but enterprise level pusher server that can push notifications
 
 ## Features
 
-- Support both APN and GCM push services with a simple unified API interface.
-- Keep persistent connections to APN following Apple recommendation.
-- Use redis pub/sub mechanism for real time push notifications. No polling.
+- EventMachine based for efficient use of resources.
+- Support both APN, GCM, WNS, and WPNS push services with a simple unified json API interface.
+- Keep persistent connections to APN following Apple's recommendations.
+- Uses the APN protocol version 2 for batched notifications.
 - No need to set APN certificates or GCM api keys in configuration files or pusher startup. These are sent in a per request basis. This allows support for multiple APN certs and GCM api keys in a single Pusher instance.
-- EventMachine based to handle in the order of thousands of push requests per second.
 
 ## Installation
 
@@ -113,15 +113,8 @@ When sending push notifications to iOS devices you must provide an aps hash insi
 
 Read the [official documentation](http://developer.apple.com/library/ios/#documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Chapters/ApplePushService.html#//apple_ref/doc/uid/TP40008194-CH100-SW1) for details on the *aps* hash format. Note that this hash must not exceed the 256 bytes or it will be rejected by the APN service.
 
-#### Sending one message to both APN and GCM
-
-Normally you would send messages to either Android or iOS indenpendently. But the pusher daemon can send the same message to devices on both networks as long as you follow the Apple restrictions. This is because Apple push messages are more limited than GCM.
-
-If your data hash is compatible with the APN standard as described above and you specify a APN cert, a GCM api_key, a list of apn_ids and a list of gcm_ids then the message will be delivered via push notification to all the devices in those lists. Apple will display the notifications using their own mechanisms and for Android you will receive the data hash in a Bundle object as usual. Is your responsibility to extract the data from that bundle and display/use it as you please.
-
-
-
 #### Sending one message to WNS
+
 message hash: { 
 		wnstype: "type" ,
 		 wnsrequeststatus: true,
@@ -136,7 +129,6 @@ secret and sid: App identification credentials provided by microsoft when regist
 wnsrequeststatus: boolean, if true, the response from wns server will have aditional information
 wnsids: jsonArray of target devices.
 
-
 #### Sending one message to WPNS
 message hash: {  wptype: "type",
 	  	 wpids: ["xxx"],
@@ -150,8 +142,6 @@ secret: a unic hash to identify a conection, internal use, each notification sen
 wpids: jsonArray of ids for target devices  
 data: notification data to send, please use de xsml template provided by Microsoft(http://msdn.microsoft.com/en-us/library/windowsphone/develop/hh202945(v=vs.105).aspx) for each wptype listed above
 
-
-
 ## Examples
 
 A simple example using ruby code to send a push notification to iOS devices.
@@ -181,11 +171,9 @@ redis = Redis.new({ host: "localhost", port: 6379})
 # Push the message to the *suj_pusher_queue* in the redis server:
 redis.lpush "pusher:suj_pusher_msgs", msg_json
 
-# Notify workers there is a new message
-redis.publish "pusher:suj_pusher_queue", "PUSH_MSG"
 ```
 
-First you must push your messages to the *suj_pusher_msgs* queue and then publish a *PUSH_MSG* to the *suj_pusher_queue*. The first is a simple queue that stores the messages and the second is a PUB/SUB queue that tells the push daemons that a new message is available. Also make sure to set the same namespace (e.g. pusher) to the queue names that you set on the pusher daemons.
+Once you push the JSON message to the *suj_pusher_msgs* queue the pusher workers will retrieve and process it.
 
 ## Issues
 

data/bin/pusher

@@ -57,6 +57,7 @@ Suj::Pusher.config.update(config)
 PusherDaemon.spawn!(
   sync_log: true,
   working_dir: Dir.pwd,
+  processes: 1,
   log_file: File.join(logdir, "pusher-worker.log"),
   pid_file: File.join(piddir, "pusher-worker.pid")
 )

data/lib/suj/pusher.rb

@@ -17,8 +17,6 @@ require 'logger'
 
 module Suj
   module Pusher
-
-    QUEUE     = "suj_pusher_queue"
     MSG_QUEUE = "suj_pusher_msgs"
   end
 end

data/lib/suj/pusher/apn_connection.rb

@@ -1,4 +1,5 @@
 require "eventmachine"
+require "iobuffer"
 
 require "base64"
 module Suj
@@ -16,6 +17,7 @@ module Suj
         6 => "Invalid topic size",
         7 => "Invalid payload size",
         8 => "Invalid token",
+        10 => "Shutdown",
         255 => "Unknown error"
       }
 
@@ -26,6 +28,7 @@ module Suj
         @options = options
         @cert_key = Digest::SHA1.hexdigest(@options[:cert])
         @cert_file = File.join(Suj::Pusher.config.certs_path, @cert_key)
+        @buffer = IO::Buffer.new
         File.open(@cert_file, "w") do |f|
           f.write @options[:cert]
         end
@@ -42,18 +45,22 @@ module Suj
 
       def deliver(data)
         begin
-          @notification = Suj::Pusher::ApnNotification.new(data)
+          @notifications = []
+          data[:apn_ids].each do |apn_id|
+            @notifications << Suj::Pusher::ApnNotification.new(data.merge({token: apn_id}))
+          end
           if ! disconnected?
             info "APN delivering data"
-            send_data(@notification.data)
-            @notification = nil
+            send_data(@notifications.join)
+            info "APN push notification sent"
+            @notifications = nil
             info "APN delivered data"
           else
             info "APN connection unavailable"
           end
         rescue Suj::Pusher::ApnNotification::PayloadTooLarge => e
           error "APN notification payload too large."
-          debug @notification.data.inspect
+          debug @notifications.join.inspect
         rescue => ex
           error "APN notification error : #{ex}"
         end
@@ -64,22 +71,34 @@ module Suj
         start_tls(@ssl_options)
       end
 
+      # Receives error data from APN servers. Each error is 6 bytes long
+      # and contains:
+      #
+      #   cmd    -> 1 byte unsigned integer that is always 8
+      #   status -> 1 byte unsigned integer that indicates the error
+      #             See ERRORS array for a list
+      #   id     -> 4 byte message ID set when the message was sent
       def receive_data(data)
-        cmd, status, id = data.unpack("ccN")
-        if status != 0
-          error "APN push error received: #{ERRORS[status]}"
-        else
-          info "APN push notification sent"
+        @buffer << data
+        while @buffer.size >= 6
+          res = @buffer.read(6)
+          cmd, status, id = data.unpack("CCN")
+          if cmd != 8
+            error "APN push response command differs from 8"
+          elsif status != 0
+            error "APN push error received: #{ERRORS[status]} for id #{id}"
+          end
         end
       end
 
       def connection_completed
         info "APN Connection established..."
         @disconnected = false
-        if ! @notification.nil?
+        if ! @notifications.nil?
           info "EST - APN delivering data"
-          send_data(@notification.data)
-          @notification = nil
+          send_data(@notifications.join)
+          info "APN push notification sent"
+          @notifications = nil
           info "EST - APN delivered data"
         end
       end
@@ -87,7 +106,6 @@ module Suj
       def unbind
         info "APN Connection closed..."
         @disconnected = true
-        FileUtils.rm_f(@cert_file)
         @pool.remove_connection(@cert_key)
       end
     end

data/lib/suj/pusher/apn_feedback_connection.rb

@@ -1,4 +1,5 @@
 require "eventmachine"
+require "iobuffer"
 
 require "base64"
 module Suj
@@ -6,12 +7,15 @@ module Suj
     class APNFeedbackConnection < EM::Connection
       include Suj::Pusher::Logger
 
-      def initialize(options = {})
+      def initialize(pool, options = {})
         super
         @disconnected = true
         @options = options
-        @cert_key = Digest::SHA1.hexdigest(@options[:cert])
+        @pool = pool
+        @cert_key = Digest::SHA1.hexdigest("FEEDBACK" + @options[:cert])
         @cert_file = File.join(Suj::Pusher.config.certs_path, @cert_key)
+        @buffer = IO::Buffer.new
+        self.comm_inactivity_timeout = 10          # Close after 10 sec of inactivity
         File.open(@cert_file, "w") do |f|
           f.write @options[:cert]
         end
@@ -31,9 +35,21 @@ module Suj
         start_tls(@ssl_options)
       end
 
+      # Receive feedback data from APN servers.
+      #
+      # The format is:
+      #
+      #   timestamp -> 4 byte bigendian
+      #   len       -> 2 byte token length
+      #   token     -> 32 bytes token
       def receive_data(data)
-        timestamp, size, token = data.unpack("QnN")
-        info "APN Feedback invalid token #{token}"
+        @buffer << data
+
+        while @buffer.size >= 38
+          timestamp, size = @buffer.read(6).unpack("Nn")
+          token = @buffer.read(size)
+          puts " FEEDBACK TIMESTAMP: #{timestamp}  SIZE: #{size} TOKEN: #{token}"
+        end
       end
 
       def connection_completed
@@ -44,6 +60,7 @@ module Suj
       def unbind
         info "APN Feedback Connection closed..."
         @disconnected = true
+        @pool.remove_connection(@cert_key)
       end
     end
   end

data/lib/suj/pusher/apn_notification.rb

@@ -23,6 +23,10 @@ module Suj
         @data ||= encode_data
       end
 
+      def to_s
+        data
+      end
+
       private
 
       def get_expiry
@@ -34,12 +38,31 @@ module Suj
       end
 
       def encode_data
-        identifier = 0
-        expiry = get_expiry
-        size = [payload].pack("a*").size
-        data_array = [1, identifier, expiry, 32, @token, size, payload]
-        info("PAYLOAD: #{data_array}")
-        data_array.pack("cNNnH*na*")
+        # identifier = 0
+        # expiry = get_expiry
+        # size = [payload].pack("a*").size
+        # data_array = [1, identifier, expiry, 32, @token, size, payload]
+        # info("PAYLOAD: #{data_array}")
+        # data_array.pack("cNNnH*na*")
+        
+        items = [
+          [1, 32,               @token ],                           # token
+          [2, payload.bytesize, payload ],                          # payload
+          [3, 4,                OpenSSL::Random::random_bytes(4) ], # random identifier
+          [4, 4,                get_expiry ],                       # expiration date
+          [5, 1,                10 ]                                # high priority
+        ]
+
+        info("PAYLOAD: #{items}")
+
+        frame_data = 
+          items[0].pack("CnH*") +
+          items[1].pack("CnA*") +
+          items[2].pack("CnA*") +
+          items[3].pack("CnN")  +
+          items[4].pack("CnC")
+
+        [2,frame_data.bytesize,frame_data].pack("CNA*")
       end
     end
   end

data/lib/suj/pusher/connection_pool.rb

@@ -1,4 +1,5 @@
 require "base64"
+require "thread"
 require File.join File.dirname(File.expand_path(__FILE__)), "apn_connection.rb"
 require File.join File.dirname(File.expand_path(__FILE__)), "gcm_connection.rb"
 
@@ -17,61 +18,77 @@ module Suj
       def initialize(daemon)
         @pool = {}
         @daemon = daemon
+        @mutex = Mutex.new
       end
 
       def apn_connection(options = {})
         cert = Digest::SHA1.hexdigest options[:cert]
         info "APN connection #{cert}"
-        @pool[cert] ||= EM.connect(APN_GATEWAY, APN_PORT, APNConnection, self, options)
+        @mutex.synchronize do
+          @pool[cert] ||= EM.connect(APN_GATEWAY, APN_PORT, APNConnection, self, options)
+        end
       end
 
       def apn_sandbox_connection(options = {})
         cert = Digest::SHA1.hexdigest options[:cert]
         info "APN connection #{cert}"
-        @pool[cert] ||= EM.connect(APN_SANDBOX, APN_PORT, APNConnection, self, options)
+        @mutex.synchronize do
+          @pool[cert] ||= EM.connect(APN_SANDBOX, APN_PORT, APNConnection, self, options)
+        end
       end
 
       def feedback_connection(options = {})
-        info "Feedback connection"
-        EM.connect(FEEDBACK_GATEWAY, FEEDBACK_PORT, APNFeedbackConnection, options)
+        cert = Digest::SHA1.hexdigest("FEEDBACK" + options[:cert])
+        info "APN Feedback connection #{cert}"
+        @mutex.synchronize do
+          @pool[cert] ||= EM.connect(FEEDBACK_GATEWAY, FEEDBACK_PORT, APNFeedbackConnection, self, options)
+        end
       end
 
       def feedback_sandbox_connection(options = {})
-        info "Feedback sandbox connection"
-        EM.connect(FEEDBACK_SANDBOX, FEEDBACK_PORT, APNFeedbackConnection, options)
+        info "Feedback connection"
+        cert = Digest::SHA1.hexdigest("FEEDBACK" + options[:cert])
+        info "APN Sandbox Feedback connection #{cert}"
+        @mutex.synchronize do
+          @pool[cert] ||= EM.connect(FEEDBACK_SANDBOX, FEEDBACK_PORT, APNFeedbackConnection, self, options)
+        end
       end
 
       def gcm_connection(options = {})
         # All GCM connections are unique, even if they are to the same app.
         api_key = "#{options[:api_key]}#{rand * 100}"
         info "GCM connection #{api_key}"
-        @pool[api_key] ||= Suj::Pusher::GCMConnection.new(self, api_key, options)
+        @mutex.synchronize do
+          @pool[api_key] ||= Suj::Pusher::GCMConnection.new(self, api_key, options)
+        end
       end
 
       def apn_connection(options = {})
         cert = Digest::SHA1.hexdigest options[:cert]
         info "APN connection #{cert}"
-        @pool[cert] ||= EM.connect(APN_GATEWAY, APN_PORT, APNConnection, self, options)
+        @mutex.synchronize do
+          @pool[cert] ||= EM.connect(APN_GATEWAY, APN_PORT, APNConnection, self, options)
+        end
       end
 
       def wns_connection(options = {})
         cert = Digest::SHA1.hexdigest options[:secret]
         info "WNS connection #{cert}"
         info "WNS Options #{options}"
-        @pool[cert] ||= Suj::Pusher::WNSConnection.new(self,options)
+        @mutex.synchronize do
+          @pool[cert] ||= Suj::Pusher::WNSConnection.new(self,options)
+        end
       end
 
       def wpns_connection(options = {})
         cert = Digest::SHA1.hexdigest options[:secret]
         info "WPNS connection #{cert}"
         info "WPNS Options #{options}"
-        @pool[cert] ||= Suj::Pusher::WPNSConnection.new(self,options)
-        return @pool[cert]
+        @mutex.synchronize do
+          @pool[cert] ||= Suj::Pusher::WPNSConnection.new(self,options)
+        end
       end
 
-
-
-
       def remove_connection(key)
         info "Removing connection #{key}"
         info "Connection not found" unless @pool.delete(key)

data/lib/suj/pusher/daemon.rb

@@ -16,22 +16,10 @@ module Suj
       def start
         info "Starting pusher daemon"
         info " subsribe to push messages from #{redis_url} namespace #{redis_namespace}"
+        @last_feedback = Time.now
+        @last_sandbox_feedback = Time.now
         EM.run do
-          wait_msg do |msg|
-            begin
-              info "RECEIVED MESSAGE"
-              data = Hash.symbolize_keys(MultiJson.load(msg))
-              send_notification(data)
-              info "SENT MESSAGE"
-              retrieve_feedback(data)
-              info "FINISHED FEEDBACK RETRIEVAL"
-            rescue MultiJson::LoadError
-              warn("Received invalid json data, discarding msg")
-            rescue => e
-              error("Error sending notification : #{e}")
-              error e.backtrace
-            end
-          end
+          wait_msg
         end
       end
 
@@ -47,28 +35,25 @@ module Suj
       private
 
       def wait_msg
-        redis.on(:connected) { info "REDIS - Connected to Redis server #{redis_url}" }
-        redis.on(:closed) { info "REDIS - Closed connection to Redis server" }
-        redis.on(:failed) { info "REDIS - redis connection FAILED" }
-        redis.on(:reconnected) { info "REDIS - Reconnected to Redis server" }
-        redis.on(:disconnected) { info "REDIS - Disconnected from Redis server" }
-        redis.on(:reconnect_failed) { info "REDIS - Reconnection attempt to Redis server FAILED" }
-        # EM.add_periodic_timer(30) { redis.publish Suj::Pusher::QUEUE, "ECHO" }
-        redis.pubsub.subscribe("#{redis_namespace}:#{Suj::Pusher::QUEUE}") do |msg|
-          if msg == "ECHO"
-            info "REDIS - ECHO Received"
-          elsif msg == "PUSH_MSG"
-            info "REDIS - PUSH_MSG Received"
-            get_message.callback do |message|
-              if message
-                yield message
-              else
-                info "REDIS - PUSH_MSG Queue was empty"
-              end
-            end
-          else
-            yield msg
+        defer = redis.brpop "#{redis_namespace}:#{MSG_QUEUE}", 0
+        defer.callback do |_, msg|
+          begin
+            info "RECEIVED MESSAGE"
+            data = Hash.symbolize_keys(MultiJson.load(msg))
+            send_notification(data)
+            info "SENT MESSAGE"
+            retrieve_feedback(data)
+          rescue MultiJson::LoadError
+            warn("Received invalid json data, discarding msg")
+          rescue => e
+            error("Error sending notification : #{e}")
+            error e.backtrace
           end
+          EM.next_tick { wait_msg }
+        end
+        defer.errback do |e|
+          error e
+          EM.next_tick { wait_msg }
         end
       end
 
@@ -105,28 +90,30 @@ module Suj
       def send_apn_notification(msg)
         info "Sending APN notification via connection #{Digest::SHA1.hexdigest(msg[:cert])}"
         conn = pool.apn_connection(msg)
-        msg[:apn_ids].each do |apn_id|
-          conn.deliver(msg.merge({token: apn_id}))
-        end
+        conn.deliver(msg)
+        # msg[:apn_ids].each do |apn_id|
+        #   conn.deliver(msg.merge({token: apn_id}))
+        # end
       end
 
       def send_apn_sandbox_notification(msg)
         info "Sending APN sandbox notification via connection #{Digest::SHA1.hexdigest(msg[:cert])}"
         conn = pool.apn_sandbox_connection(msg)
-        msg[:apn_ids].each do |apn_id|
-          conn.deliver(msg.merge({token: apn_id}))
-        end
+        conn.deliver(msg)
+        # msg[:apn_ids].each do |apn_id|
+        #   conn.deliver(msg.merge({token: apn_id}))
+        # end
       end
 
       def feedback_connection(msg)
-        return if @last_feedback and (Time.now - @last_feedback < FEEDBACK_TIME)
+        return Time.now - @last_feedback < FEEDBACK_TIME
         info "Get feedback information"
         conn = pool.feedback_connection(msg)
         @last_feedback = Time.now
       end
 
       def feedback_sandbox_connection(msg)
-        return if @last_sandbox_feedback and (Time.now - @last_sandbox_feedback < FEEDBACK_TIME)
+        return if Time.now - @last_sandbox_feedback < FEEDBACK_TIME
         info "Get feedback sandbox information"
         conn = pool.feedback_sandbox_connection(msg)
         @last_sandbox_feedback = Time.now
@@ -150,8 +137,6 @@ module Suj
         conn.deliver(msg)
       end
 
-
-
       def redis_url
         @redis_url ||= "redis://#{Suj::Pusher.config.redis_host}:#{Suj::Pusher.config.redis_port}/#{Suj::Pusher.config.redis_db}"
       end
@@ -161,14 +146,41 @@ module Suj
       end
 
       def redis
-        @redis ||= EM::Hiredis.connect(redis_url)
-      end
+        return @redis if ! @redis.nil?
 
-      def get_message
-        @redis_connection ||= EM::Hiredis.connect(redis_url)
-        @redis_connection.rpop "#{redis_namespace}:#{MSG_QUEUE}"
+        @redis = EM::Hiredis.connect(redis_url)
+
+        @redis.on(:connected) { info "REDIS - Connected to Redis server #{redis_url}" }
+
+        @redis.on(:closed) do
+          info "REDIS - Closed connection to Redis server"
+          @redis = nil
+        end
+
+        @redis.on(:failed) do
+          info "REDIS - redis connection FAILED"
+          @redis = nil
+        end
+
+        @redis.on(:reconnected) { info "REDIS - Reconnected to Redis server" }
+
+        @redis.on(:disconnected) do
+          info "REDIS - Disconnected from Redis server"
+          @redis = nil
+        end
+
+        @redis.on(:reconnect_failed) do
+          info "REDIS - Reconnection attempt to Redis server FAILED"
+          @redis = nil
+        end
+
+        return @redis
       end
 
+      # def get_message
+      #   @redis_connection ||= EM::Hiredis.connect(redis_url)
+      # end
+
       def pool
         @pool ||= Suj::Pusher::ConnectionPool.new(self)
       end

data/lib/suj/pusher/version.rb

@@ -1,5 +1,5 @@
 module Suj
   module Pusher
-    VERSION = '0.2.0'
+    VERSION = '0.2.3'
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: suj-pusher
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.2.3
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-10-31 00:00:00.000000000 Z
+date: 2013-12-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: em-http-request
-  requirement: &16244860 !ruby/object:Gem::Requirement
+  requirement: &10281040 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *16244860
+  version_requirements: *10281040
 - !ruby/object:Gem::Dependency
   name: em-hiredis
-  requirement: &16244440 !ruby/object:Gem::Requirement
+  requirement: &10280620 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *16244440
+  version_requirements: *10280620
 - !ruby/object:Gem::Dependency
   name: multi_json
-  requirement: &16244020 !ruby/object:Gem::Requirement
+  requirement: &10280020 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,10 +43,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *16244020
+  version_requirements: *10280020
 - !ruby/object:Gem::Dependency
   name: daemon-spawn
-  requirement: &16243600 !ruby/object:Gem::Requirement
+  requirement: &10279380 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,7 +54,18 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *16243600
+  version_requirements: *10279380
+- !ruby/object:Gem::Dependency
+  name: iobuffer
+  requirement: &10295080 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *10295080
 description: Stand alone push notification server for APN, GCM, WNS and MPNS.
 email:
 - rd@skillupjapan.co.jp, n.bersano@skillupchile.cl