hookshot 0.3.0 → 0.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 50a6ddf79d002197181d01e453820959a59c24fd
-  data.tar.gz: 2af7e0bb07e0d6810cb2e1e8722f44b97ea891c0
+  metadata.gz: 50a5202fd2cf12d2977095bd54a10af1284bf61b
+  data.tar.gz: 094bb40322e173db3a428db8bfdb61b40c7a7388
 SHA512:
-  metadata.gz: 6af6ecbb0d55181f235d3b9a9c7f79124e3c29e575bb0856dee3d038b1d8c5375119014906f0d5640d89dfddde2bf6d6024ae4de2fcd0e9274d9ec89ef8a7edc
-  data.tar.gz: bd380b8d21b96d091698e5fdc35e86193a8b334cd268fcee7e86e5f71d43b3ebc124683ea6b5a1549cdfeffb91854316e5667e7b75299fd89b14ad2104ad499b
+  metadata.gz: 6233b77583a4bb4ea005f2d6b7e43b5fa1503edca65e1af206ce3507a489de1e3866b8a3f3d817c902a79eaab9bee3510f9e8560f0554448e47a421c09fede65
+  data.tar.gz: 3d3b2b0aee201ec4bfedb319f19db5a409c0667bc6c548ba39cb6825e21ae2933d5f0f08643c41fb91a86d985e5c8343238ed576a86471a67a80c4547a62ffd6

data/README.md

@@ -1,29 +1,114 @@
-# Hookshot::Client
+# Hookshot
 
-TODO: Write a gem description
+Hookshot is a rubygem for submitting jobs to the
+[Hookshot](https://github.com/Shopify/hookshot) webhook delivery engine.
 
 ## Installation
 
-Add this line to your application's Gemfile:
+Hookshot is on rubygems, just add the latest version to your gemfile. Setup of
+the `hookshot` server is separate and more complex, so [go read about that
+first](https://github.com/shopify/hookshot).
 
-    gem 'hookshot'
+## Usage
 
-And then execute:
+#### Initialize a connection
 
-    $ bundle
+```ruby
+require 'hookshot'
+require 'redis'
+hookshot = Hookshot.new(Redis.new(port: 6379, host: 'localhost'))
+```
 
-Or install it yourself as:
+#### Enqueue a webhook
 
-    $ gem install hookshot
+```ruby
+hookshot.enqueue(
+  url: 'http://localhost:8080/post',
+  headers: {"X-My-Header" => "value"},
+  context: "42",
+  payload: "request body")
+```
 
-## Usage
+#### Enqueue a webhook with a delay
+
+You can enqueue a job but have it activate only after a specified delay.
+
+```ruby
+hookshot.enqueue_in(60, # seconds
+  url: 'http://localhost:8080/post',
+  headers: {"X-My-Header" => "value"},
+  context: "42",
+  payload: "request body")
+```
+
+#### Read back failed jobs
+
+Jobs that fail many times in a row are returned back to the application.
+Specifically, the `context` value passed in via the `enqueue*` methods is
+returned.
+
+`get_next_failure` returns two values: the number of failures so far for this
+job, and the `context` passed in with the job. If `nfailures` is equal to
+`Hookshot::FINAL_FAILURE`, the job will not be retried. However, if `nfailures`
+is any other value, the job will still be retried; this is just an advisory
+because the job has been failing for at least 24 hours.
+
+```ruby
+loop {
+  nfailures, context = hookshot.get_next_failure
+  if nfailures == Hookshot::FINAL_FAILURE
+    delete_webhook_subscription(context)
+  end
+}
+```
+
+#### Check queue stats
+
+Hookshot writes a lot of statistics to statsd/datadog, but to quickly check the
+current queue sizes, use `queue_stats`.
+
+```ruby
+hookshot.queue_stats
+# => { pending: 42, delayed: 42, failures: 42 }
+```
+
+### Blacklist and Whitelist
+
+Hookshot provides a manual blacklist and a manual whitelist:
+
+* Jobs for any domain in the blacklist are automatically rejected by hookshot.
+* Throttles for any domain in the whitelist are automatically allocated the maximum allowed throughput.
+
+The format of the domain should be the full domain. The port should not be included if it is `80` or `443`.
+
+Examples:
+
+| URL                       | Domain              |
+|---------------------------|---------------------|
+| http://example.com/post   | example.com         |
+| https://example.com/post  | example.com         |
+| http://example.com:8000   | example.com:8000    |
+| http://example.com:80     | example.com         |
+
+#### Check the lists
+
+```ruby
+hookshot.blacklist
+#=> ["example.com"]
+hookshot.whitelist
+#=> ["google.com", "shopify.com"]
+```
+
+#### Add an item to the list
 
-TODO: Write usage instructions here
+```ruby
+hookshot.blacklist!("example.com")
+hookshot.whitelist!("google.com")
+```
 
-## Contributing
+#### Remove an item from the list
 
-1. Fork it
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+````ruby
+hookshot.remove_blacklist("example.com")
+hookshot.remove_whitelist("google.com")
+```

data/lib/hookshot.rb

@@ -9,15 +9,57 @@ class Hookshot
   NEW_JOBS_LIST    = "#{PREFIX}:jobs"
   DELAYED_SET      = "#{PREFIX}:delayed"
   FAILURES_LIST    = "#{PREFIX}:failures"
-  MANUAL_BLACKLIST = "#{PREFIX}:throttle:blacklist"
+  BLACKLIST        = "#{PREFIX}:throttle:blacklist"
+  WHITELIST        = "#{PREFIX}:throttle:whitelist"
 
+  # Special sentinel value returned from `get_next_failure` when the job will no
+  # longer be retried.
   FINAL_FAILURE = -1
 
+  # Jobs are retried for up to 48 hours. Though we delete the job info when
+  # hookshot is done with each job, it's still a good idea to clean up any keys
+  # that have managed to stick around somehow.
+  JOB_KEY_LIFETIME = 86400 * 4
+
+  # Raised by `get_next_failure` if there are no pending failures.
+  FailureQueueEmpty = Class.new(StandardError)
+
   attr_reader :redis
+
+  # Hookshot expects to be initialized with an instance of `Redis` provided by
+  # the `redis` rubygem.
+  #
+  # Example:
+  #
+  #  require 'hookshot'
+  #  require 'redis'
+  #  hookshot = Hookshot.new(Redis.new(port: 6379, host: 'localhost'))
   def initialize(redis)
     @redis = redis
   end
 
+  # enqueue takes a URL, a hash of headers, a payload (request body), and a
+  # `context` value. It submits these values to hookshot for processing.
+  #
+  # The `context` value can be any string, and will be returned to you via
+  # `get_next_failure` if the job can't be successfully completed after about 48
+  # hours of retries.
+  #
+  # In Shopify, we pass our `WebhookSubscription` object ID for the `context`
+  # value, so that we can notify merchants when their webhooks are failing, and
+  # delete subscriptions that fail consistently.
+  #
+  # `activate_at` is an optional parameter that specifies a number of seconds to
+  # wait before making this job active in hookshot. You should normally call
+  # `enqueue_in` rather than pass `activate_at` explicitly.
+  #
+  # Example:
+  #
+  #   hookshot.enqueue(
+  #     url: 'http://localhost:8080/post',
+  #     headers: {"X-My-Header" => "value"},
+  #     context: "42",
+  #     payload: "request body")
   def enqueue(url:, headers:, context:, payload:, activate_at: nil)
     uuid = SecureRandom.uuid
     redis.pipelined do
@@ -28,6 +70,7 @@ class Hookshot
         "context", context,
         "payload", payload,
         "failures", 0)
+      redis.expire(job_key(uuid), JOB_KEY_LIFETIME)
       if activate_at
         redis.zadd(DELAYED_SET, activate_at, uuid)
       else
@@ -38,6 +81,16 @@ class Hookshot
     uuid
   end
 
+  # enqeueue_in calls enqueue with an `activate_at` parameter to delay the job's
+  # execution.
+  #
+  # Example:
+  #
+  #   hookshot.enqueue_in(60, # seconds
+  #     url: 'http://localhost:8080/post',
+  #     headers: {"X-My-Header" => "value"},
+  #     context: "42",
+  #     payload: "request body")
   def enqueue_in(duration, url:, headers:, context:, payload:)
     enqueue_time = (Time.now + duration).to_i
     enqueue(
@@ -48,9 +101,32 @@ class Hookshot
       activate_at: enqueue_time)
   end
 
+  # Jobs that fail many times in a row are returned back to the application.
+  # Specifically, the `context` value passed in via the `enqueue*` methods is
+  # returned.
+  #
+  # `get_next_failure` returns two values: the number of failures so far for
+  # this job, and the `context` passed in with the job. If `nfailures` is equal
+  # to `Hookshot::FINAL_FAILURE`, the job will not be retried. However, if
+  # `nfailures` is any other value, the job will still be retried; this is just
+  # an advisory because the job has been failing for at least 24 hours.
+  #
+  # This method is non-blocking: if there is no item present in the failures
+  # queue, it will raise FailureQueueEmpty.
+  #
+  # Example:
+  #
+  #   loop {
+  #     nfailures, context = hookshot.get_next_failure
+  #     if nfailures == Hookshot::FINAL_FAILURE
+  #       delete_webhook_subscription(context)
+  #     end
+  #   }
   def get_next_failure
     # block indefinitely waiting for the next failure.
-    line = redis.blpop(FAILURES_LIST, 0)
+    line = redis.lpop(FAILURES_LIST)
+    raise FailureQueueEmpty if line.nil?
+
     nfailures, failed_id = line.split('|', 2)
 
     if nfailures.to_i == 0 || failed_id.empty?
@@ -60,14 +136,13 @@ class Hookshot
     [nfailures.to_i, failed_id]
   end
 
-  def force_blacklist(domain, duration)
-    redis.zadd(MANUAL_BLACKLIST, Time.now.to_i+duration, domain)
-  end
-
-  def remove_override(domain)
-    redis.zrem(MANUAL_BLACKLIST, domain)
-  end
-
+  # Hookshot writes a lot of statistics to statsd/datadog, but to quickly check the
+  # current queue sizes, use `queue_stats`.
+  #
+  # Example:
+  #
+  #  hookshot.queue_stats
+  #  # => { pending: 42, delayed: 42, failures: 42 }
   def queue_stats
     pending, delayed, failures = redis.pipelined do
       redis.llen NEW_JOBS_LIST
@@ -77,6 +152,88 @@ class Hookshot
     { pending: pending, delayed: delayed, failures: failures }
   end
 
+  # blacklist! adds a domain to the list of currently blacklisted domains. Any
+  # jobs submitted with a domain in this list will be automatically dropped by
+  # hookshot.
+  #
+  # The format of the domain should be the full domain. The port should not be
+  # included if it is `80` or `443`. For example:
+  #
+  # | URL                       | Domain              |
+  # |---------------------------|---------------------|
+  # | http://example.com/post   | example.com         |
+  # | https://example.com/post  | example.com         |
+  # | http://example.com:8000   | example.com:8000    |
+  # | http://example.com:80     | example.com         |
+  #
+  # Example:
+  #
+  #   hookshot.blacklist!("example.com")
+  def blacklist!(domain)
+    redis.sadd(BLACKLIST, domain)
+  end
+
+  # whitelist! adds a domain to the list of currently whitelisted domains.
+  # Normally, jobs are throttled to a maximum requests per second per domain.
+  # Whitelisted domains are granted a much higher initial rate.
+  #
+  # The format of the domain should be the full domain. The port should not be
+  # included if it is `80` or `443`. For example:
+  #
+  # | URL                       | Domain              |
+  # |---------------------------|---------------------|
+  # | http://example.com/post   | example.com         |
+  # | https://example.com/post  | example.com         |
+  # | http://example.com:8000   | example.com:8000    |
+  # | http://example.com:80     | example.com         |
+  #
+  # Example:
+  #
+  #   hookshot.whitelist!("google.com")
+  def whitelist!(domain)
+    redis.sadd(WHITELIST, domain)
+  end
+
+  # blacklist returns an array of currently-blacklisted domains.
+  #
+  # Example:
+  #
+  #   hookshot.blacklist
+  #   #=> ["example.com"]
+  def blacklist
+    redis.smembers(BLACKLIST)
+  end
+
+  # whitelist returns an array of currently-whitelisted domains.
+  #
+  # Example:
+  #
+  #   hookshot.whitelist
+  #   #=> ["example.com"]
+  def whitelist
+    redis.smembers(WHITELIST)
+  end
+
+  # remove_blacklist removes a currently-blacklisted domain from the blacklist.
+  # If the domain was not blacklisted, this method has no effect.
+  #
+  # Example:
+  #
+  #   hookshot.remove_blacklist("google.com")
+  def remove_blacklist(domain)
+    redis.srem(BLACKLIST, domain)
+  end
+
+  # remove_whitelist removes a currently-whitelisted domain from the whitelist.
+  # If the domain was not whitelisted, this method has no effect.
+  #
+  # Example:
+  #
+  #   hookshot.remove_whitelist("google.com")
+  def remove_whitelist(domain)
+    redis.srem(WHITELIST, domain)
+  end
+
   private
 
   def serialize_headers(headers)

data/lib/hookshot/version.rb

@@ -1,3 +1,3 @@
 class Hookshot
-  VERSION = "0.3.0"
+  VERSION = "0.3.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hookshot
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.3.2
 platform: ruby
 authors:
 - Burke Libbey
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-09-11 00:00:00.000000000 Z
+date: 2014-09-12 00:00:00.000000000 Z
 dependencies: []
 description: Hookshot client library for ruby
 email: