prepor-protobuf 3.5.0

data/CONTRIBUTING.md

@@ -0,0 +1,16 @@
+# Contributing
+
+I love accepting issues and pull requests. I only ask for a few guidelines to
+be followed that will make it much easier for me to solve your issue or get
+your code merged.
+
+1. Use GitHub Issues or Pull Requests over sending an email. It's much easier for me to keep track of your issue through GitHub.
+2. For __compiler issues__, please provide both a gist for the __source definition(s)__ as well as the __generated output__ (if any).
+3. For __existing issues or functionality__, please use __`2-8-stable`__ as the base branch for the pull request. This helps us maintain a stable gem release strategy. All commits merged to `2-8-stable` will also be merged down to `master`.
+4. For __new functionality__, please use __`master`__ as the base branch for the pull request. The `master` branch is used to stage all "next iteration" work.
+5. Be patient with me as I work on your issue.
+
+Following these simple guidelines really will help me help you. And really,
+that's what we're here for. I'm on @localshred on twitter, let's be friends. :)
+
+## Happy Contributing!

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright (c) 2014 BJ Neilsen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+

data/README.md

@@ -0,0 +1,33 @@
+# protobuf
+
+[![Gem Version](https://badge.fury.io/rb/protobuf.svg)](http://badge.fury.io/rb/protobuf)
+[![Build Status](https://secure.travis-ci.org/localshred/protobuf.svg?branch=master)](https://travis-ci.org/localshred/protobuf)
+[![Gitter chat](https://badges.gitter.im/localshred/protobuf.svg)](https://gitter.im/localshred/protobuf)
+[![protobuf API Documentation](https://www.omniref.com/ruby/gems/protobuf.png)](https://www.omniref.com/ruby/gems/protobuf)
+
+Protobuf is an implementation of [Google's protocol buffers][google-pb] in ruby, version 2.5.0 is currently supported.
+
+## Install
+
+See our [Installation Guide][] on the [wiki][].
+
+## Usage
+
+The [wiki][] contains in-depth guides on the various ways to use this gem
+including [compiling definitions][], [object APIs][], [services][], [clients][], and even
+an [API roadmap][].
+
+## Changelog
+
+See recent changes in the [release notes][] or the [changelog][].
+
+  [google-pb]:             http://code.google.com/p/protobuf "Google Protocol Buffers"
+  [wiki]:                  https://github.com/localshred/protobuf/wiki "Wiki home page"
+  [Installation Guide]:    https://github.com/localshred/protobuf/wiki/Installation "Installation guide"
+  [compiling definitions]: https://github.com/localshred/protobuf/wiki/Compiling-Definitions "Compiling guide"
+  [object APIs]:           https://github.com/localshred/protobuf/wiki/Messages-&-Enums "Message & Enum object APIs guide"
+  [services]:              https://github.com/localshred/protobuf/wiki/Services "Services object API guide"
+  [clients]:               https://github.com/localshred/protobuf/wiki/Clients "Client object API guide"
+  [API roadmap]:           https://github.com/localshred/protobuf/wiki/API-Roadmap "API Roadmap guide"
+  [release notes]:         https://github.com/localshred/protobuf/releases "Release notes"
+  [changelog]:             https://github.com/localshred/protobuf/blob/master/CHANGES.md "CHANGES.md"

data/Rakefile

@@ -0,0 +1,64 @@
+$LOAD_PATH << ::File.expand_path('../', __FILE__)
+$LOAD_PATH << ::File.expand_path('../spec', __FILE__)
+
+require 'fileutils'
+require 'rubygems'
+require 'rubygems/package_task'
+require 'bundler/gem_tasks'
+require 'benchmark/tasks'
+
+require 'rspec/core/rake_task'
+require 'rubocop/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+
+task :default => ['compile:spec', 'compile:rpc', :spec, :rubocop]
+
+desc 'Run specs'
+namespace :compile do
+
+  desc 'Compile spec protos in spec/supprt/ directory'
+  task :spec do
+    proto_path = ::File.expand_path('../spec/support/', __FILE__)
+    proto_files = Dir[File.join(proto_path, '**', '*.proto')]
+    cmd = %(protoc --plugin=./bin/protoc-gen-ruby --ruby_out=#{proto_path} -I #{proto_path} #{proto_files.join(' ')})
+
+    puts cmd
+    system(cmd)
+  end
+
+  desc 'Compile rpc protos in protos/ directory'
+  task :rpc do
+    proto_path = ::File.expand_path('../proto', __FILE__)
+    proto_files = Dir[File.join(proto_path, '**', '*.proto')]
+    output_dir = ::File.expand_path('../tmp/rpc', __FILE__)
+    ::FileUtils.mkdir_p(output_dir)
+
+    cmd = %(protoc --plugin=./bin/protoc-gen-ruby --ruby_out=#{output_dir} -I #{proto_path} #{proto_files.join(' ')})
+
+    puts cmd
+    system(cmd)
+
+    files = {
+      'tmp/rpc/dynamic_discovery.pb.rb'               => 'lib/protobuf/rpc',
+      'tmp/rpc/rpc.pb.rb'                             => 'lib/protobuf/rpc',
+      'tmp/rpc/google/protobuf/descriptor.pb.rb'      => 'lib/protobuf/descriptors/google/protobuf',
+      'tmp/rpc/google/protobuf/compiler/plugin.pb.rb' => 'lib/protobuf/descriptors/google/protobuf/compiler',
+    }
+
+    files.each_pair do |source_file, destination_dir|
+      source_file = ::File.expand_path("../#{source_file}", __FILE__)
+      destination_dir = ::File.expand_path("../#{destination_dir}", __FILE__)
+      ::FileUtils::Verbose.cp(source_file, destination_dir)
+    end
+  end
+
+end
+
+task :console do
+  require 'pry'
+  require 'protobuf'
+  ARGV.clear
+  ::Pry.start
+end

data/bin/protoc-gen-ruby

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+
+# Before requiring protobuf, ensure that we will not load any
+# server or client code.
+#
+ENV['PB_NO_NETWORKING'] = '1'
+
+$LOAD_PATH << ::File.expand_path("../../lib", __FILE__)
+require 'protobuf'
+require 'protobuf/descriptors'
+require 'protobuf/code_generator'
+
+request_bytes = STDIN.read
+code_generator = ::Protobuf::CodeGenerator.new(request_bytes)
+response_bytes = code_generator.response_bytes
+STDOUT.print(response_bytes)

data/bin/rpc_server

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require 'protobuf/cli'
+::Protobuf::CLI.start(ARGV)
+exit

data/install-protobuf.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env sh
+
+set -ex
+
+gdie() {
+    echo "$@" >&2
+    exit 1
+}
+
+test -n "$PROTOBUF_VERSION" || die "PROTOBUF_VERSION env var is undefined"
+
+case "$PROTOBUF_VERSION" in
+2*)
+    basename=protobuf-$PROTOBUF_VERSION
+    ;;
+3*)
+    basename=protobuf-cpp-$PROTOBUF_VERSION
+    ;;
+*)
+    die "unknown protobuf version: $PROTOBUF_VERSION"
+    ;;
+esac
+
+curl -sL https://github.com/google/protobuf/releases/download/v$PROTOBUF_VERSION/$basename.tar.gz | tar zx
+
+cd protobuf-$PROTOBUF_VERSION
+
+./configure --prefix=/usr && make -j2 && make install

data/lib/protobuf.rb

@@ -0,0 +1,100 @@
+require 'logger'
+require 'socket'
+require 'pp'
+require 'stringio'
+
+require 'active_support/core_ext/object/blank'
+require 'active_support/core_ext/object/try'
+require 'active_support/inflector'
+require 'active_support/json'
+require 'active_support/notifications'
+
+require 'protobuf/deprecation'
+
+module Protobuf
+
+  # See Protobuf#connector_type documentation.
+  CONNECTORS = [:socket, :zmq].freeze
+
+  # Default is Socket as it has no external dependencies.
+  DEFAULT_CONNECTOR = :socket
+
+  module_function
+
+  class << self
+    # Client Host
+    #
+    # Default: `hostname` of the system
+    #
+    # The name or address of the host to use during client RPC calls.
+    attr_accessor :client_host
+  end
+
+  self.client_host = Socket.gethostname
+
+  # Connector Type
+  #
+  # Default: socket
+  #
+  # Symbol value which denotes the type of connector to use
+  # during client requests to an RPC server.
+  def self.connector_type
+    @connector_type ||= DEFAULT_CONNECTOR
+  end
+
+  def self.connector_type=(type)
+    fail ArgumentError, 'Invalid connector type given' unless CONNECTORS.include?(type)
+    @connector_type = type
+  end
+
+  # GC Pause during server requests
+  #
+  # Default: false
+  #
+  # Boolean value to tell the server to disable
+  # the Garbage Collector when handling an rpc request.
+  # Once the request is completed, the GC is enabled again.
+  # This optomization provides a huge boost in speed to rpc requests.
+  def self.gc_pause_server_request?
+    return @gc_pause_server_request unless @gc_pause_server_request.nil?
+    self.gc_pause_server_request = false
+  end
+
+  def self.gc_pause_server_request=(value)
+    @gc_pause_server_request = !!value
+  end
+
+  # Permit unknown field on Message initialization
+  #
+  # Default: true
+  #
+  # Simple boolean to define whether we want to permit unknown fields
+  # on Message intialization; otherwise a ::Protobuf::FieldNotDefinedError is thrown.
+  def self.ignore_unknown_fields?
+    !defined?(@ignore_unknown_fields) || @ignore_unknown_fields
+  end
+
+  def self.ignore_unknown_fields=(value)
+    @ignore_unknown_fields = !!value
+  end
+end
+
+unless ENV.key?('PB_NO_NETWORKING')
+  require 'protobuf/rpc/client'
+  require 'protobuf/rpc/service'
+
+  env_connector_type = ENV.fetch('PB_CLIENT_TYPE') do
+    ::Protobuf::DEFAULT_CONNECTOR
+  end.to_s.downcase.strip.to_sym
+
+  if ::Protobuf::CONNECTORS.include?(env_connector_type)
+    require "protobuf/#{env_connector_type}"
+  else
+    $stderr.puts <<-WARN
+    [WARNING] Require attempted on an invalid connector type '#{env_connector_type}'.
+              Falling back to default '#{::Protobuf::DEFAULT_CONNECTOR}' connector.
+    WARN
+
+    require "protobuf/#{::Protobuf::DEFAULT_CONNECTOR}"
+  end
+end

data/lib/protobuf/cli.rb

@@ -0,0 +1,242 @@
+require 'active_support/core_ext/hash/keys'
+
+require 'thor'
+require 'protobuf/version'
+require 'protobuf/logging'
+require 'protobuf/rpc/servers/socket_runner'
+require 'protobuf/rpc/servers/zmq_runner'
+
+module Protobuf
+  class CLI < ::Thor
+    include ::Thor::Actions
+    include ::Protobuf::Logging
+
+    attr_accessor :runner, :mode, :exit_requested
+
+    no_commands do
+      alias_method :exit_requested?, :exit_requested
+    end
+
+    default_task :start
+
+    desc 'start APP_FILE', 'Run the RPC server in the given mode, preloading the given APP_FILE. This is the default task.'
+
+    option :host,                       :type => :string, :default => '127.0.0.1', :aliases => %w(-o), :desc => 'Host to bind.'
+    option :port,                       :type => :numeric, :default => 9399, :aliases => %w(-p), :desc => 'Master Port to bind.'
+
+    option :backlog,                    :type => :numeric, :default => 100, :aliases => %w(-b), :desc => 'Backlog for listening socket when using Socket Server.'
+    option :threshold,                  :type => :numeric, :default => 100, :aliases => %w(-t), :desc => 'Multi-threaded Socket Server cleanup threshold.'
+    option :threads,                    :type => :numeric, :default => 5, :aliases => %w(-r), :desc => 'Number of worker threads to run. Only applicable in --zmq mode.'
+
+    option :log,                        :type => :string, :default => STDOUT, :aliases => %w(-l), :desc => 'Log file or device. Default is STDOUT.'
+    option :level,                      :type => :numeric, :default => ::Logger::INFO, :aliases => %w(-v), :desc => 'Log level to use, 0-5 (see http://www.ruby-doc.org/stdlib/libdoc/logger/rdoc/)'
+
+    option :socket,                     :type => :boolean, :aliases => %w(-s), :desc => 'Socket Mode for server and client connections.'
+    option :zmq,                        :type => :boolean, :aliases => %w(-z), :desc => 'ZeroMQ Socket Mode for server and client connections.'
+
+    option :beacon_interval,            :type => :numeric, :desc => 'Broadcast beacons every N seconds. (default: 5)'
+    option :beacon_port,                :type => :numeric, :desc => 'Broadcast beacons to this port (default: value of ServiceDirectory.port)'
+    option :broadcast_beacons,          :type => :boolean, :desc => 'Broadcast beacons for dynamic discovery (Currently only available with ZeroMQ).'
+    option :broadcast_busy,             :type => :boolean, :default => false, :desc => 'Remove busy nodes from cluster when all workers are busy (Currently only available with ZeroMQ).'
+    option :debug,                      :type => :boolean, :default => false, :aliases => %w(-d), :desc => 'Debug Mode. Override log level to DEBUG.'
+    option :gc_pause_request,           :type => :boolean, :default => false, :desc => 'Enable/Disable GC pause during request.'
+    option :print_deprecation_warnings, :type => :boolean, :default => nil, :desc => 'Cause use of deprecated fields to be printed or ignored.'
+    option :workers_only,               :type => :boolean, :default => false, :desc => "Starts process with only workers (no broker/frontend is started) only relevant for Zmq Server"
+    option :worker_port,                :type => :numeric, :default => nil, :desc => "Port for 'backend' where workers connect (defaults to port + 1)"
+    option :zmq_inproc,                 :type => :boolean, :default => true, :desc => 'Use inproc protocol for zmq Server/Broker/Worker'
+
+    def start(app_file)
+      debug_say('Configuring the rpc_server process')
+
+      configure_logger
+      configure_traps
+      configure_runner_mode
+      create_runner
+      configure_process_name(app_file)
+      configure_gc
+      configure_deprecation_warnings
+
+      require_application(app_file) unless exit_requested?
+      start_server unless exit_requested?
+    rescue => e
+      say_and_exit('ERROR: RPC Server failed to start.', e)
+    end
+
+    desc 'version', 'Print ruby and protoc versions and exit.'
+    def version
+      say("Ruby Protobuf v#{::Protobuf::VERSION}")
+    end
+
+    no_tasks do
+
+      # Tell protobuf how to handle the printing of deprecated field usage.
+      def configure_deprecation_warnings
+        if options.print_deprecation_warnings.nil?
+          ::Protobuf.print_deprecation_warnings = !ENV.key?("PB_IGNORE_DEPRECATIONS")
+        else
+          ::Protobuf.print_deprecation_warnings = options.print_deprecation_warnings?
+        end
+      end
+
+      # If we pause during request we don't need to pause in serialization
+      def configure_gc
+        debug_say('Configuring gc')
+
+        if defined?(JRUBY_VERSION)
+          # GC.enable/disable are noop's on Jruby
+          ::Protobuf.gc_pause_server_request = false
+        else
+          ::Protobuf.gc_pause_server_request = options.gc_pause_request?
+        end
+      end
+
+      # Setup the protobuf logger.
+      def configure_logger
+        debug_say('Configuring logger')
+
+        log_level = options.debug? ? ::Logger::DEBUG : options.level
+
+        ::Protobuf::Logging.initialize_logger(options.log, log_level)
+
+        # Debug output the server options to the log file.
+        logger.debug { 'Debugging options:' }
+        logger.debug { options.inspect }
+      end
+
+      # Re-write the $0 var to have a nice process name in ps.
+      def configure_process_name(app_file)
+        debug_say('Configuring process name')
+        $0 = "rpc_server --#{mode} #{options.host}:#{options.port} #{app_file}"
+      end
+
+      # Configure the mode of the server and the runner class.
+      def configure_runner_mode
+        debug_say('Configuring runner mode')
+
+        self.mode = if multi_mode?
+          say('WARNING: You have provided multiple mode options. Defaulting to socket mode.', :yellow)
+          :socket
+        elsif options.zmq?
+          :zmq
+        else # rubocop:disable Style/ElseAlignment
+          # above: https://github.com/bbatsov/rubocop/pull/1470/files
+          case server_type = ENV["PB_SERVER_TYPE"]
+          when nil, /socket/i
+            :socket
+          when /zmq/i
+            :zmq
+          else
+            say "WARNING: You have provided incorrect option 'PB_SERVER_TYPE=#{server_type}'. Defaulting to socket mode.", :yellow
+            :socket
+          end
+        end
+      end
+
+      # Configure signal traps.
+      # TODO: add signal handling for hot-reloading the application.
+      def configure_traps
+        debug_say('Configuring traps')
+
+        exit_signals = [:INT, :TERM]
+        exit_signals << :QUIT unless defined?(JRUBY_VERSION)
+
+        exit_signals.each do |signal|
+          debug_say("Registering trap for exit signal #{signal}", :blue)
+
+          trap(signal) do
+            self.exit_requested = true
+            shutdown_server
+          end
+        end
+      end
+
+      # Create the runner for the configured mode
+      def create_runner
+        debug_say("Creating #{mode} runner")
+        self.runner = case mode
+        when :zmq
+          create_zmq_runner
+        when :socket
+          create_socket_runner
+        else
+          say_and_exit("Unknown runner mode: #{mode}")
+        end
+      end
+
+      # Say something if we're in debug mode.
+      def debug_say(message, color = :yellow)
+        say(message, color) if options.debug?
+      end
+
+      # Internal helper to determine if the modes are multi-set which is not valid.
+      def multi_mode?
+        options.zmq? && options.socket?
+      end
+
+      # Require the application file given, exiting if the file doesn't exist.
+      def require_application(app_file)
+        debug_say('Requiring app file')
+        require app_file
+      rescue LoadError => e
+        say_and_exit("Failed to load application file #{app_file}", e)
+      end
+
+      def runner_options
+        opt = options.symbolize_keys
+
+        opt[:workers_only] = (!!ENV['PB_WORKERS_ONLY']) || options.workers_only
+
+        opt
+      end
+
+      def say_and_exit(message, exception = nil)
+        message = set_color(message, :red) if options.log == STDOUT
+
+        logger.error { message }
+
+        if exception
+          $stderr.puts "[#{exception.class.name}] #{exception.message}"
+          $stderr.puts exception.backtrace.join("\n")
+
+          logger.error { "[#{exception.class.name}] #{exception.message}" }
+          logger.debug { exception.backtrace.join("\n") }
+        end
+
+        exit(1)
+      end
+
+      def create_socket_runner
+        require 'protobuf/socket'
+
+        self.runner = ::Protobuf::Rpc::SocketRunner.new(runner_options)
+      end
+
+      def create_zmq_runner
+        require 'protobuf/zmq'
+
+        self.runner = ::Protobuf::Rpc::ZmqRunner.new(runner_options)
+      end
+
+      def shutdown_server
+        logger.info { 'RPC Server shutting down...' }
+        runner.stop
+        ::Protobuf::Rpc::ServiceDirectory.instance.stop
+      end
+
+      # Start the runner and log the relevant options.
+      def start_server
+        debug_say('Running server')
+
+        runner.run do
+          logger.info do
+            "pid #{::Process.pid} -- #{mode} RPC Server listening at #{options.host}:#{options.port}"
+          end
+
+          ::ActiveSupport::Notifications.instrument("after_server_bind")
+        end
+
+        logger.info { 'Shutdown complete' }
+      end
+    end
+  end
+end