thrifter 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: fd99b7900e55cc7abcfe145198cb184b4dd27d28
-  data.tar.gz: b5aeb9adef656c411b43327a619496f0dd46b429
+  metadata.gz: d17b3472039110ba6cb2226fb72e0ed490f3a0a9
+  data.tar.gz: 0dc804e6e5de0d3e3107961d5841ca834db39d29
 SHA512:
-  metadata.gz: 30d3d385ef740047a5dae0b0f704518a5e9c77e551bd29d7cf991a65b33fb5255a4b246571e6088ae36a457bb5ca37a9d90dda2c949efd50f3bfcb7431ce09be
-  data.tar.gz: c0c644e11697991a4f2187dd505906a53de4829d74f7653b7f65ef25965ddf6649a78caca0f5d5e05a64ab9385d805f6c981a6cadbc380c43b898391f73711e0
+  metadata.gz: ce63b7eae7e10ad4caa724be4ef43082b0673ebe54c734cd95f3a273fbae579b694769b9c377ad3671fd33b77dd31d7b5d81da15bc84d9399a20ed8df9f45d11
+  data.tar.gz: 7d8f048ee242cf2dcdf03a4091f0ed3694e6b5a14fdaf0fed0019031ab0aee03cde8d0a2cc566921879c9a70ea4c7002452238fb2e14df704633d1a0fafb5e7d

data/README.md

@@ -81,37 +81,19 @@ class MyClient < Thrifer.build(MyService::Client)
   config.uri = 'tcp://foo:2383'
 end
 
-#The common block form is supported as well
+# The common block form is supported as well
 MyClient.configure do |config|
   # ... same as above
 end
 ```
 
-### RPC Metrics with Statsd
+### Extensions
 
-Statsd metrics are **opt-in**. By default, `Thrifter` sets the statsd
-client to a null implementation. If you want metrics, set
-`config.statsd` to an object that implements the [statsd-ruby][]
-interface. `Thrifter` emits the following metrics:
-
-* time on each rpc calls
-* number of `Thrift::TransportException`
-* number of `Thrift::ProtocolExeption`
-* number of `Thrift::ApplicationExeption`
-* number of `Timeout::Error`
-* number of generic errors (e.g. none of the above known errors)
-
-It's recommended that the `statsd` object do namespacing
-(statsd-ruby has it built in). This ensures client metrics don't
-get intermingled with wider application metrics. Here's an example:
-
-```ruby
-ServiceClient = Thrifter.build(MyService::Client)
-# Now in production.rb
-ServiceClient.config.statsd = Statsd.new namespace: 'my_service'
-```
+Extensions add functionality to the client itself. They do not effect
+the request/response cycle in anyway. `Thrifter` includes a few
+extensions by default. This section covers each included extension.
 
-### RPC Queuing
+#### Queuing
 
 Certain systems may need to queue RPCs to other systems. This is only
 useful for `void` RPCs or for when an outside system may be flaky.
@@ -119,8 +101,13 @@ Assume `MyService` has a `logStats` RPC. Your application is producing
 stats that should make it upstream, but there are intermitent network
 problems effeciting stats collection. Include `Thrift::Queueing` and
 any RPC will automatically be sent to sidekiq for eventual processing.
+`sidekiq` must be available. This is an **opt-in** dependency, of if
+you want this funtionality, add `sidekiq` and
+`sidekiq-thrift_arguments` to your `Gemfile`.
 
 ```ruby
+require 'thrifter/extensions/queueing'
+
 class ServiceClient < Thrifter.build(MyService::Client)
   include Thrifter::Queuing
 end
@@ -144,7 +131,7 @@ end
 All RPCs will be sent to the `thrift` sidekiq queue. They will follow
 default sidekiq retry backoff and the like.
 
-### RPC Retrying
+#### Retry Support
 
 Systems have syncrhonous RPCs. Unfortunately sometimes these don't
 work for whatever reason. It's good practice to retry these RPCs
@@ -178,13 +165,40 @@ end
 Look into something like [retriable][] if you want a more robust
 solution for different use cases.
 
+#### Pinging
+
+Components in a system may need to inquire if other systems are
+available before continuing. `Thrifer::Ping` is just that.
+`Thrifter::Ping` assumes the service has a `ping` RPC. If your
+service does not have one (or is named differently) simply implement
+the `ping` method on the class. Any successful response will count as
+up, anything else will not.
+
+```ruby
+class MyService < Thrifer.build(MyService::Client)
+  include Thrifer::Ping
+
+  # Define a ping method if the service does not have one
+  def ping
+    my_other_rpc
+  end
+end
+
+# my_service.up? # => true
+```
+
 ### Middleware
 
 The middleware approach is great for providing a flexible extension
 points to hook into the RPC process. `Thrifter::Client` provides a
 middleware implementation to common to many other ruby libraries.
-Middleware can only be customized at the class level. This ensures
-middleware applies when used in extensions.
+Unlike extensions, middleware modify the request/response cycle. They
+do not modify the client class directly. `Thrifter` includes a few
+helpful middlware which are documented below.
+
+#### Using Middleware
+
+Here's the most simple middlware example:
 
 ```ruby
 class MyClient < Thrifter.build(MyService::Client)
@@ -218,7 +232,31 @@ class LoggingMiddleware
 end
 ```
 
-### Error Wrapping
+#### Metrics
+
+Statsd metrics are **opt-in**. By default, `Thrifter` sets the statsd
+client to a null implementation. If you want metrics, set
+`config.statsd` to an object that implements the [statsd-ruby][]
+interface. `Thrifter` emits the following metrics:
+
+* time on each rpc calls
+* number of `Thrift::TransportException`
+* number of `Thrift::ProtocolExeption`
+* number of `Thrift::ApplicationExeption`
+* number of `Timeout::Error`
+* number of generic errors (e.g. none of the above known errors)
+
+It's recommended that the `statsd` object do namespacing
+(statsd-ruby has it built in). This ensures client metrics don't
+get intermingled with wider application metrics. Here's an example:
+
+```ruby
+ServiceClient = Thrifter.build(MyService::Client)
+# Now in production.rb
+ServiceClient.config.statsd = Statsd.new namespace: 'my_service'
+```
+
+#### Error Wrapping
 
 A lot of things can go wrong in the thrift stack. This means the
 caller may need to deal with a large amount of different exceptions.
@@ -249,26 +287,22 @@ end
 Note, `Thrifter` will still count individual errors as described in
 the metrics section.
 
-### Pinging
+#### Protocol Validation
 
-Components in a system may need to inquire if other systems are
-available before continuing. `Thrifer::Ping` is just that.
-`Thrifter::Ping` assumes the service has a `ping` RPC. If your
-service does not have one (or is named differently) simply implement
-the `ping` method on the class. Any successful response will count as
-up, anything else will not.
+Thrift requires that client & server communicate with the exact things
+specified in the protocol. Unfortunately ruby does not prevent us from
+making mistakes. It's possible to forget setting required members or
+assigning symbol instead of a string. Luckily ruby's dynamic traits
+make it possible to implement compiler like validation. `Thrifter`
+includes a middlware that will checkout incoming & outgoing objects so
+that they're valid protocol message. [thrift-validator][] does all the
+heavy lifing here. Use `Thrifter::Validation` in the test
+environment to make sure things are correct. Here's an example.
 
 ```ruby
-class MyService < Thrifer.build(MyService::Client)
-  include Thrifer::Ping
-
-  # Define a ping method if the service does not have one
-  def ping
-    my_other_rpc
-  end
+class MyService < Thrifter.build(MyService::Client)
+  use Thrifter::Validation
 end
-
-# my_service.up? # => true
 ```
 
 ## Contributing

data/lib/thrifter.rb

@@ -1,11 +1,13 @@
 require 'thrifter/version'
 
 require 'forwardable'
+require 'delegate'
 require 'uri'
 require 'tnt'
 require 'concord'
 require 'thrift'
 require 'thrift-base64'
+require 'thrift-validator'
 require 'middleware'
 require 'connection_pool'
 
@@ -84,6 +86,14 @@ module Thrifter
       end
     end
 
+    class DirectDispatcher
+      include Concord.new(:app, :client)
+
+      def call(rpc)
+        client.send rpc.name, *rpc.args
+      end
+    end
+
     class << self
       extend Forwardable
 
@@ -115,12 +125,14 @@ module Thrifter
       end
     end
 
-    def initialize
-      fail ArgumentError, 'config.uri not set!' unless config.uri
+    def initialize(client = nil)
+      if client.nil?
+        fail ArgumentError, 'config.uri not set!' unless config.uri
 
-      uri = URI(config.uri)
+        uri = URI(config.uri)
 
-      fail ArgumentError, 'URI did not contain port' unless uri.port
+        fail ArgumentError, 'URI did not contain port' unless uri.port
+      end
 
       @pool = ConnectionPool.new size: config.pool_size.to_i, timeout: config.pool_timeout.to_f do
         stack = MiddlewareStack.new
@@ -130,13 +142,17 @@ module Thrifter
         # Insert metrics here so metrics are as close to the network
         # as possible. This excludes time in any middleware an
         # application may have configured.
-        stack.use StatsdMiddleware, config.statsd
+        stack.use Metrics, config.statsd
 
-        socket = Thrift::Socket.new uri.host, uri.port, config.rpc_timeout.to_f
-        transport = config.transport.new socket
-        protocol = config.protocol.new transport
+        if client.nil?
+          socket = Thrift::Socket.new uri.host, uri.port, config.rpc_timeout.to_f
+          transport = config.transport.new socket
+          protocol = config.protocol.new transport
 
-        stack.use Dispatcher, transport, client_class.new(protocol)
+          stack.use Dispatcher, transport, client_class.new(protocol)
+        else
+          stack.use DirectDispatcher, client
+        end
 
         stack.finalize!
       end
@@ -160,7 +176,9 @@ module Thrifter
   end
 end
 
-require_relative 'thrifter/statsd_middleware'
-require_relative 'thrifter/ping'
-require_relative 'thrifter/error_wrapping_middleware'
-require_relative 'thrifter/retry'
+require_relative 'thrifter/extensions/ping'
+require_relative 'thrifter/extensions/retriable'
+
+require_relative 'thrifter/middleware/error_wrapping'
+require_relative 'thrifter/middleware/validation'
+require_relative 'thrifter/middleware/metrics'

data/lib/thrifter/{error_wrapping_middleware.rb → middleware/error_wrapping.rb}

@@ -3,7 +3,7 @@ module Thrifter
     "#{ex.class}: #{ex.message}"
   end
 
-  class ErrorWrappingMiddleware
+  class ErrorWrapping
     WRAP = [
       Thrift::TransportException,
       Thrift::ProtocolException,

data/lib/thrifter/{statsd_middleware.rb → middleware/metrics.rb}

@@ -1,5 +1,5 @@
 module Thrifter
-  class StatsdMiddleware
+  class Metrics
     include Concord.new(:app, :statsd)
 
     def call(rpc)

data/lib/thrifter/middleware/validation.rb

@@ -0,0 +1,34 @@
+module Thrifter
+  ValidationError = Tnt.boom do |rpc, ex|
+    "Invalid data in RPC #{rpc.name}! #{ex.message}"
+  end
+
+  class Validation
+    include Concord.new(:app)
+
+    def call(rpc)
+      validate rpc, rpc.args
+      response = app.call rpc
+
+      validate rpc, Array(response)
+
+      response
+    end
+
+    private
+
+    def validate(rpc, objects)
+      objects.each do |item|
+        if item.is_a? Thrift::Struct
+          begin
+            Thrift::Validator.new.validate(item)
+          rescue Thrift::ProtocolException => ex
+            raise ValidationError.new(rpc, ex)
+          end
+        else
+          next
+        end
+      end
+    end
+  end
+end

data/lib/thrifter/version.rb

@@ -1,3 +1,3 @@
 module Thrifter
-  VERSION = "0.1.0"
+  VERSION = "0.1.1"
 end

data/script/monkey-client

@@ -11,7 +11,7 @@ end
 
 require 'bundler/setup'
 require 'thrifter'
-require 'thrifter/queueing'
+require 'thrifter/extensions/queueing'
 require 'securerandom'
 
 require 'eventmachine'

data/test/acceptance_test.rb

@@ -190,6 +190,26 @@ class AcceptanceTest < MiniTest::Unit::TestCase
     assert_match /testing/, result.message, 'Middleware not called'
   end
 
+  def test_implementation_maybe_given_at_instantation_time
+    implementation = stub echo: :stubbed_response
+
+    client = Thrifter.build TestClient
+    client.config.uri = uri
+
+    thrifter = client.new implementation
+    assert_equal :stubbed_response, thrifter.echo(:request)
+  end
+
+  def test_does_not_require_uri_when_providing_an_implementation
+    implementation = stub echo: :stubbed_response
+
+    client = Thrifter.build TestClient
+    client.config.uri = nil
+
+    thrifter = client.new implementation
+    assert_equal :stubbed_response, thrifter.echo(:request)
+  end
+
   def test_close_the_transport_on_successful_rpc
     transport = mock
     client = Thrifter.build TestClient

data/test/{ping_test.rb → extensions/ping_test.rb}

@@ -1,4 +1,4 @@
-require_relative './test_helper'
+require_relative '../test_helper'
 
 class PingTest < MiniTest::Unit::TestCase
   def test_returns_false_if_anything_goes_wrong

data/test/{queuing_test.rb → extensions/queueing_test.rb}

@@ -1,4 +1,4 @@
-require_relative 'test_helper'
+require_relative '../test_helper'
 
 class QueuingTest < MiniTest::Unit::TestCase
   class TestClient < TestService::Client

data/test/{retry_test.rb → extensions/retry_test.rb}

@@ -1,4 +1,4 @@
-require_relative 'test_helper'
+require_relative '../test_helper'
 
 class RetryTest < MiniTest::Unit::TestCase
   JunkError = Class.new StandardError

data/test/{error_wrapping_middleware_test.rb → middleware/error_wrapping_test.rb}

@@ -1,12 +1,12 @@
-require_relative './test_helper'
+require_relative '../test_helper'
 
-class ErrorWrappingMiddlewareTest < MiniTest::Unit::TestCase
+class ErrorWrappingTest < MiniTest::Unit::TestCase
   TestError = Class.new StandardError
 
   attr_reader :rpc
 
   def known_errors
-    Thrifter::ErrorWrappingMiddleware.wrapped
+    Thrifter::ErrorWrapping.wrapped
   end
 
   def setup
@@ -19,7 +19,7 @@ class ErrorWrappingMiddlewareTest < MiniTest::Unit::TestCase
     app = stub
     app.stubs(:call).with(rpc).raises(known_errors.first)
 
-    middleware = Thrifter::ErrorWrappingMiddleware.new app
+    middleware = Thrifter::ErrorWrapping.new app
 
     assert_raises Thrifter::ClientError do
       middleware.call rpc
@@ -30,7 +30,7 @@ class ErrorWrappingMiddlewareTest < MiniTest::Unit::TestCase
     app = stub
     app.stubs(:call).with(rpc).raises(TestError)
 
-    middleware = Thrifter::ErrorWrappingMiddleware.new app, [ TestError ]
+    middleware = Thrifter::ErrorWrapping.new app, [ TestError ]
 
     assert_raises Thrifter::ClientError do
       middleware.call rpc
@@ -41,7 +41,7 @@ class ErrorWrappingMiddlewareTest < MiniTest::Unit::TestCase
     app = stub
     app.stubs(:call).with(rpc).raises(TestError.new('testing 123'))
 
-    middleware = Thrifter::ErrorWrappingMiddleware.new app, [ TestError ]
+    middleware = Thrifter::ErrorWrapping.new app, [ TestError ]
 
     error = assert_raises Thrifter::ClientError do
       middleware.call rpc

data/test/{statsd_middleware_test.rb → middleware/metrics_test.rb}

@@ -1,6 +1,6 @@
-require_relative './test_helper'
+require_relative '../test_helper'
 
-class StatsdMiddlewareTest < MiniTest::Unit::TestCase
+class MetricsTest < MiniTest::Unit::TestCase
   attr_reader :rpc
 
   def setup
@@ -16,7 +16,7 @@ class StatsdMiddlewareTest < MiniTest::Unit::TestCase
     statsd = mock
     statsd.expects(:time).with(rpc.name).yields.returns(:response)
 
-    middleware = Thrifter::StatsdMiddleware.new app, statsd
+    middleware = Thrifter::Metrics.new app, statsd
     result = middleware.call rpc
 
     assert :response == result, 'Return value incorrect'
@@ -30,7 +30,7 @@ class StatsdMiddlewareTest < MiniTest::Unit::TestCase
     statsd.expects(:time).yields
     statsd.expects(:increment).with('errors.transport')
 
-    middleware = Thrifter::StatsdMiddleware.new app, statsd
+    middleware = Thrifter::Metrics.new app, statsd
 
     assert_raises Thrift::TransportException do
       middleware.call rpc
@@ -45,7 +45,7 @@ class StatsdMiddlewareTest < MiniTest::Unit::TestCase
     statsd.expects(:time).yields
     statsd.expects(:increment).with('errors.protocol')
 
-    middleware = Thrifter::StatsdMiddleware.new app, statsd
+    middleware = Thrifter::Metrics.new app, statsd
 
     assert_raises Thrift::ProtocolException do
       middleware.call rpc
@@ -60,7 +60,7 @@ class StatsdMiddlewareTest < MiniTest::Unit::TestCase
     statsd.expects(:time).yields
     statsd.expects(:increment).with('errors.application')
 
-    middleware = Thrifter::StatsdMiddleware.new app, statsd
+    middleware = Thrifter::Metrics.new app, statsd
 
     assert_raises Thrift::ApplicationException do
       middleware.call rpc
@@ -75,7 +75,7 @@ class StatsdMiddlewareTest < MiniTest::Unit::TestCase
     statsd.expects(:time).yields
     statsd.expects(:increment).with('errors.timeout')
 
-    middleware = Thrifter::StatsdMiddleware.new app, statsd
+    middleware = Thrifter::Metrics.new app, statsd
 
     assert_raises Timeout::Error do
       middleware.call rpc
@@ -90,7 +90,7 @@ class StatsdMiddlewareTest < MiniTest::Unit::TestCase
     statsd.expects(:time).yields
     statsd.expects(:increment).with('errors.other')
 
-    middleware = Thrifter::StatsdMiddleware.new app, statsd
+    middleware = Thrifter::Metrics.new app, statsd
 
     assert_raises StandardError do
       middleware.call rpc

data/test/middleware/validation_test.rb

@@ -0,0 +1,57 @@
+require_relative '../test_helper'
+
+class ValidationTest < MiniTest::Unit::TestCase
+  attr_reader :rpc, :invalid_struct, :valid_struct
+
+  def setup
+    super
+
+    @rpc = Thrifter::RPC.new :echo
+
+    @invalid_struct = TestMessage.new
+    @valid_struct = TestMessage.new message: 'testing'
+  end
+
+  def test_fails_if_args_contains_invalid_structs
+    rpc.args = [ invalid_struct ]
+
+    client = stub call: valid_struct
+
+    middleware = Thrifter::Validation.new client
+
+    assert_raises Thrifter::ValidationError do
+      middleware.call rpc
+    end
+  end
+
+  def test_does_not_validate_primitive_request_values
+    rpc.args = [ 1, 2, 3 ]
+
+    client = stub call: valid_struct
+
+    middleware = Thrifter::Validation.new client
+
+    assert_equal valid_struct, middleware.call(rpc)
+  end
+
+  def test_fails_if_repsonse_contains_invalid_structs
+    rpc.args = [ valid_struct ]
+
+    client = stub call: invalid_struct
+
+    middleware = Thrifter::Validation.new client
+
+    assert_raises Thrifter::ValidationError do
+      middleware.call rpc
+    end
+  end
+
+  def test_does_not_validate_primitive_response_values
+    rpc.args = [ valid_struct ]
+
+    client = stub call: [ 1, 2 ]
+
+    middleware = Thrifter::Validation.new client
+    assert_equal [ 1,2 ], middleware.call(rpc)
+  end
+end

data/test/test_helper.rb

@@ -1,6 +1,6 @@
 require 'bundler/setup'
 require 'thrifter'
-require 'thrifter/queueing'
+require 'thrifter/extensions/queueing'
 
 root = File.expand_path '../..', __FILE__
 $LOAD_PATH << "#{root}/vendor/gen-rb"

data/thrifter.gemspec

@@ -20,6 +20,7 @@ Gem::Specification.new do |spec|
 
   spec.add_dependency "thrift"
   spec.add_dependency "thrift-base64"
+  spec.add_dependency "thrift-validator"
   spec.add_dependency "statsd-ruby"
   spec.add_dependency "concord"
   spec.add_dependency "middleware"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: thrifter
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.1
 platform: ruby
 authors:
 - ahawkins
@@ -38,6 +38,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: thrift-validator
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: statsd-ruby
   requirement: !ruby/object:Gem::Requirement
@@ -210,21 +224,23 @@ files:
 - Vagrantfile
 - circle.yml
 - lib/thrifter.rb
-- lib/thrifter/error_wrapping_middleware.rb
-- lib/thrifter/ping.rb
-- lib/thrifter/queueing.rb
-- lib/thrifter/retry.rb
-- lib/thrifter/statsd_middleware.rb
+- lib/thrifter/extensions/ping.rb
+- lib/thrifter/extensions/queueing.rb
+- lib/thrifter/extensions/retriable.rb
+- lib/thrifter/middleware/error_wrapping.rb
+- lib/thrifter/middleware/metrics.rb
+- lib/thrifter/middleware/validation.rb
 - lib/thrifter/version.rb
 - script/monkey-client
 - script/server
 - test.thrift
 - test/acceptance_test.rb
-- test/error_wrapping_middleware_test.rb
-- test/ping_test.rb
-- test/queuing_test.rb
-- test/retry_test.rb
-- test/statsd_middleware_test.rb
+- test/extensions/ping_test.rb
+- test/extensions/queueing_test.rb
+- test/extensions/retry_test.rb
+- test/middleware/error_wrapping_test.rb
+- test/middleware/metrics_test.rb
+- test/middleware/validation_test.rb
 - test/test_helper.rb
 - thrifter.gemspec
 - vendor/gen-rb/test_constants.rb
@@ -256,9 +272,10 @@ specification_version: 4
 summary: Production ready Thrift client with improved semantics
 test_files:
 - test/acceptance_test.rb
-- test/error_wrapping_middleware_test.rb
-- test/ping_test.rb
-- test/queuing_test.rb
-- test/retry_test.rb
-- test/statsd_middleware_test.rb
+- test/extensions/ping_test.rb
+- test/extensions/queueing_test.rb
+- test/extensions/retry_test.rb
+- test/middleware/error_wrapping_test.rb
+- test/middleware/metrics_test.rb
+- test/middleware/validation_test.rb
 - test/test_helper.rb