subserver 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ed0057cefa344204a08c579efd92a9f0df33aae3
+  data.tar.gz: 1d76ebad4bcf2f65e7dda873a21a39c13cebcd6c
+SHA512:
+  metadata.gz: 07745b6c86ef5001384bf36fa6195f01ffa8c73676c4012936ae3d33a174eedad014336a3b94b2873b931eafec3b2fa56af4659234266f11a943a57799a72a77
+  data.tar.gz: 8ca91c9f88a06a5ab50ba7290da4754bf6f98fbfdb7ce6798054f31e359f431673dc1006d5c07222c642907f88f2172ad32158d466d6c97444f59faaa88cc7c8

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+# 1.0.0
+- Initial Public Release

data/CONTRIBUTING.md

@@ -0,0 +1,54 @@
+# Contribution Guide
+
+Subserver is one of Life.Church's Open Source projects as part of our [opendigerati](https://www.opendigerati.com/) initiative. 
+Because of this we are continuously trying to improve the process for contributing to our projects. 
+This document outlines our current process, but please check back here every time you wish to contribute. 
+
+## Open Development
+All work on Subserver happens directly on GitHub. Both Life.Church team members and external contributors send pull requests which go through the same review process.
+
+## Branch Organization
+We will do our best to keep the master branch in good shape, with tests passing at all times. But in order to move fast, we will make API changes that your application might not be compatible with. We recommend that you use the latest stable version of Subserver in your application.
+If you send a pull request, please do it against the master branch. We maintain stable branches for major versions separately but we don’t accept pull requests to them directly. Instead, we cherry-pick non-breaking changes from master to the latest stable major version.
+
+## Semantic Versioning
+Subserver follows [semantic versioning](http://semver.org/). We release patch versions for bugfixes, minor versions for new features, and major versions for any breaking changes. When we make breaking changes, we also introduce deprecation warnings in a minor version so that our users learn about the upcoming changes and migrate their code in advance.
+
+We will add a version milestone to every approved pull request marking whether the change should go in the next patch, minor, or a major version. 
+
+Every significant change is documented in the [changelog](./CHANGELOG.md) file.
+
+## Bugs
+### Where to Find Known Issues
+We are using GitHub Issues for our public bugs. We keep a close eye on this and try to make it clear when we have an internal fix in progress. Before filing a new task, try to make sure your problem doesn’t already exist.
+
+### Reporting New Issues
+When reporting new issues, please be as detailed as possible and provide the following:
+* Version of Subserver
+* Version of Ruby
+* Version of Rails (If using)
+* Steps to Reproduce the Bug
+* Any steps you have tried to resolve the issue
+
+### Security Bugs
+For bugs dealing with security vulnerabilities please **do not** post a public issue. Please email the code maintainers [@wintheday](https://github.com/wintheday).
+
+### Proposing a Change
+If you intend to change the API, or make any non-trivial changes to the implementation, we recommend filing an issue. This lets us reach an agreement on your proposal before you put significant effort into it.
+If you’re only fixing a bug, it’s fine to submit a pull request right away but we still recommend to file an issue detailing what you’re fixing. This is helpful in case we don’t accept that specific fix but want to keep track of the issue.
+
+### Your First Pull Request
+If this is your first ever Pull Request we are honored you have chosen to join Open Source through our community. You can learn the basics of Github and how Pull Requests work from this free video series:
+[How to Contribute to an Open Source Project on GitHub](https://egghead.io/series/how-to-contribute-to-an-open-source-project-on-github)
+
+### Sending a Pull Request
+The code maintainers are monitoring for pull requests. We will review your pull request and either merge it, request changes to it, or close it with an explanation. For breaking changes we may need to fix our internal uses of Validstate, which could cause some delay. We’ll do our best to provide updates and feedback throughout the process.
+
+#### Before submitting a pull request, please make sure the following is done:
+1. Fork the repository and create your branch from master.
+2. Run `bundle install` from the repository root.
+3. If you’ve fixed a bug or added code that should be tested, add tests!
+4. Ensure the test suite passes `rspec`. 
+
+### Making changes to the Documentation 
+Please ensure that any change made to Subserver that introduce new functionality or a breaking change to the API are well documented via the wiki. 

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2018 Life.Church
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,5 @@
+Subserver
+==============
+
+<!-- [![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq) -->
+[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](./CONTRIBUTING.md)

data/bin/subserver

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+
+$TESTING = false
+
+require_relative '../lib/subserver/cli'
+
+begin
+  cli = Subserver::CLI.instance
+  cli.parse
+  cli.run
+rescue => e
+  raise e if $DEBUG
+  STDERR.puts e.message
+  STDERR.puts e.backtrace.join("\n")
+  exit 1
+end

data/lib/subserver.rb

@@ -0,0 +1,185 @@
+require 'json'
+
+require 'subserver/version'
+fail "Subserver #{Subserver::VERSION} does not support Ruby versions below 2.3.1." if RUBY_PLATFORM != 'java' && Gem::Version.new(RUBY_VERSION) < Gem::Version.new('2.3.1')
+
+require 'subserver/logging'
+require 'subserver/pubsub'
+require 'subserver/health'
+require 'subserver/subscriber'
+require 'subserver/middleware/chain'
+
+module Subserver
+  NAME = 'Subserver'
+  LICENSE = 'Subserver is licensed under MIT.'
+
+  DEFAULTS = {
+    project_id: nil,
+    credentials: nil,
+    queues: [],
+    labels: [],
+    require: '.',
+    subscriber_dir: './subscribers',
+    environment: nil,
+    timeout: 35,
+    error_handlers: [],
+    death_handlers: [],
+    lifecycle_events: {
+      startup: [],
+      quiet: [],
+      shutdown: [],
+      heartbeat: [],
+    },
+    reloader: proc { |&block| block.call },
+  }
+
+  DEFAULT_SUBSCRIBER_OPTIONS = {
+    subscription: nil,
+    deadline: 60,
+    streams: 2, 
+    threads: {
+      callback: 4,
+      push: 2
+    },
+    inventory: 1000,
+    queue: 'default'
+  }
+
+
+  def self.options
+    @options ||= load_config
+  end
+
+  def self.options=(opts)
+    @options = opts
+  end
+
+  def self.configure
+    yield self
+  end
+
+  def self.load_config(file=nil)
+    opts = DEFAULTS.dup
+    file = Dir["config/subserver.yml*"].first if file.nil?
+    return opts unless file && File.exists?(file)
+    opts.merge(parse_config(file))
+  end
+
+  def self.load_json(string)
+    JSON.parse(string)
+  end
+
+  def self.dump_json(object)
+    JSON.generate(object)
+  end
+
+  def self.logger
+    Subserver::Logging.logger
+  end
+
+  def self.logger=(log)
+    Subserver::Logging.logger = log
+  end
+
+  def self.health_server
+    @health_server ||= Subserver::Health.new
+  end
+
+  def self.pubsub_client
+    Subserver::Pubsub.client
+  end
+
+  def self.middleware
+    @chain ||= default_middleware
+    yield @chain if block_given?
+    @chain
+  end
+
+  def self.default_middleware
+    Middleware::Chain.new
+  end
+
+  def self.default_subscriber_options=(hash)
+    @default_subscriber_options = default_subscriber_options.merge(Hash[hash.map{|k, v| [k.to_s, v]}])
+  end
+
+  def self.default_subscriber_options
+    defined?(@default_subscriber_options) ? @default_subscriber_options : DEFAULT_SUBSCRIBER_OPTIONS
+  end
+
+  # Death handlers are called when all retries for a job have been exhausted and
+  # the job dies.  It's the notification to your application
+  # that this job will not succeed without manual intervention.
+  #
+  # Subserver.configure do |config|
+  #   config.death_handlers << ->(job, ex) do
+  #   end
+  # end
+  def self.death_handlers
+    options[:death_handlers]
+  end
+
+  # Register a proc to handle any error which occurs within the Subserver process.
+  #
+  #   Subserver.configure do |config|
+  #     config.error_handlers << proc {|ex,ctx_hash| MyErrorService.notify(ex, ctx_hash) }
+  #   end
+  #
+  # The default error handler logs errors to Subserver.logger.
+  def self.error_handlers
+    self.options[:error_handlers]
+  end
+
+  # Register a block to run at a point in the Subserver lifecycle.
+  # :startup, :quiet or :shutdown are valid events.
+  #
+  #   Subserver.configure do |config|
+  #     config.on(:shutdown) do
+  #       puts "Goodbye cruel world!"
+  #     end
+  #   end
+  def self.on(event, &block)
+    raise ArgumentError, "Symbols only please: #{event}" unless event.is_a?(Symbol)
+    raise ArgumentError, "Invalid event name: #{event}" unless options[:lifecycle_events].key?(event)
+    options[:lifecycle_events][event] << block
+  end
+
+  # We are shutting down Subserver but what about workers that
+  # are working on some long job?  This error is
+  # raised in workers that have not finished within the hard
+  # timeout limit.  This is needed to rollback db transactions,
+  # otherwise Ruby's Thread#kill will commit.
+  # DO NOT RESCUE THIS ERROR IN YOUR SUBSCRIBERS
+  class Shutdown < Interrupt; end
+
+  private
+  
+  def self.parse_config(cfile)
+    opts = {}
+    if File.exist?(cfile)
+      opts = YAML.load(ERB.new(IO.read(cfile)).result) || opts
+
+      if opts.respond_to? :deep_symbolize_keys!
+        opts.deep_symbolize_keys!
+      else
+        symbolize_keys_deep!(opts)
+      end
+
+    else
+      # allow a non-existent config file so Subserver
+      # can be deployed by cap with just the defaults.
+    end
+    opts
+  end
+
+  def self.symbolize_keys_deep!(hash)
+    hash.keys.each do |k|
+      symkey = k.respond_to?(:to_sym) ? k.to_sym : k
+      hash[symkey] = hash.delete k
+      symbolize_keys_deep! hash[symkey] if hash[symkey].kind_of? Hash
+    end
+  end
+
+end
+
+require 'subserver/rails' if defined?(::Rails::Engine)

data/lib/subserver/cli.rb

@@ -0,0 +1,353 @@
+# frozen_string_literal: true
+$stdout.sync = true
+
+require 'yaml'
+require 'singleton'
+require 'optparse'
+require 'erb'
+require 'fileutils'
+
+require 'subserver'
+require 'subserver/util'
+
+module Subserver
+  class CLI
+    include Util
+    include Singleton unless $TESTING
+
+    attr_accessor :code
+    attr_accessor :launcher
+    attr_accessor :environment
+
+    def initialize
+      @code = nil
+    end
+
+    def parse(args=ARGV)
+      @code = nil
+
+      setup_options(args)
+      initialize_logger
+      validate!
+      daemonize
+      write_pid
+    end
+
+    def jruby?
+      defined?(::JRUBY_VERSION)
+    end
+
+    def run
+      boot_system
+      print_banner
+
+      self_read, self_write = IO.pipe
+      sigs = %w(INT TERM TTIN TSTP)
+      # USR1 and USR2 don't work on the JVM
+      if !jruby?
+        sigs << 'USR1'
+        sigs << 'USR2'
+      end
+
+      sigs.each do |sig|
+        begin
+          trap sig do
+            self_write.write("#{sig}\n")
+          end
+        rescue ArgumentError
+          puts "Signal #{sig} not supported"
+        end
+      end
+
+      logger.info "Running in #{RUBY_DESCRIPTION}"
+      logger.info Subserver::LICENSE
+
+      # cache process identity
+      Subserver.options[:identity] = identity
+
+      # Touch middleware so it isn't lazy loaded by multiple threads.
+      Subserver.middleware
+
+      # Before this point, the process is initializing with just the main thread.
+      # Starting here the process will now have multiple threads running.
+      fire_event(:startup, reverse: false, reraise: true)
+
+      logger.debug { "Middleware: #{Subserver.middleware.map(&:klass).join(', ')}" }
+
+      if !options[:daemon]
+        logger.info 'Starting processing, hit Ctrl-C to stop'
+      end
+
+      # Start Health Server
+      @health_thread = safe_thread("health_server") do
+        Subserver.health_server.start
+      end 
+
+      require 'subserver/launcher'
+      @launcher = Subserver::Launcher.new(options)
+
+      begin
+        launcher.run
+
+        while readable_io = IO.select([self_read])
+          signal = readable_io.first[0].gets.strip
+          handle_signal(signal)
+        end
+      rescue Interrupt
+        logger.info 'Shutting down'
+        launcher.stop
+        exit(0)
+      end
+    end
+
+    def self.banner
+%q{
+================================
+           Subserver
+================================
+}
+    end
+
+    SIGNAL_HANDLERS = {
+      # Ctrl-C in terminal
+      'INT' => ->(cli) { raise Interrupt },
+      # TERM is the signal that Subserver must exit.
+      # Heroku sends TERM and then waits 30 seconds for process to exit.
+      'TERM' => ->(cli) { raise Interrupt },
+      'USR1' => ->(cli) {
+        Subserver.logger.info "Received USR1, no longer accepting new work"
+        cli.launcher.quiet
+      },
+      'TSTP' => ->(cli) {
+        Subserver.logger.info "Received TSTP, no longer accepting new work"
+        cli.launcher.quiet
+      },
+      'USR2' => ->(cli) {
+        if Subserver.options[:logfile]
+          Subserver.logger.info "Received USR2, reopening log file"
+          Subserver::Logging.reopen_logs
+        end
+      },
+      'TTIN' => ->(cli) {
+        Thread.list.each do |thread|
+          Subserver.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread['subserver_label']}"
+          if thread.backtrace
+            Subserver.logger.warn thread.backtrace.join("\n")
+          else
+            Subserver.logger.warn "<no backtrace available>"
+          end
+        end
+      },
+    }
+
+    def handle_signal(sig)
+      Subserver.logger.debug "Got #{sig} signal"
+      handy = SIGNAL_HANDLERS[sig]
+      if handy
+        handy.call(self)
+      else
+        Subserver.logger.info { "No signal handler for #{sig}" }
+      end
+    end
+
+    private unless $TESTING
+
+    def print_banner
+      # Print logo and banner for development
+      if environment == 'development' && $stdout.tty?
+        puts Subserver::CLI.banner
+      end
+    end
+
+    def daemonize
+      return unless options[:daemon]
+
+      raise ArgumentError, "You really should set a logfile if you're going to daemonize" unless options[:logfile]
+      files_to_reopen = []
+      ObjectSpace.each_object(File) do |file|
+        files_to_reopen << file unless file.closed?
+      end
+
+      ::Process.daemon(true, true)
+
+      files_to_reopen.each do |file|
+        begin
+          file.reopen file.path, "a+"
+          file.sync = true
+        rescue ::Exception
+        end
+      end
+
+      [$stdout, $stderr].each do |io|
+        File.open(options[:logfile], 'ab') do |f|
+          io.reopen(f)
+        end
+        io.sync = true
+      end
+      $stdin.reopen('/dev/null')
+
+      initialize_logger
+    end
+
+    def set_environment(cli_env)
+      @environment = cli_env || ENV['RAILS_ENV'] || ENV['RACK_ENV'] || 'development'
+    end
+
+    alias_method :die, :exit
+    alias_method :☠, :exit
+
+    def setup_options(args)
+      opts = parse_options(args)
+      set_environment opts[:environment]
+
+      cfile = opts[:config_file]
+      opts = Subserver.load_config(cfile).merge(opts)
+
+      Subserver.options = opts
+    end
+
+    def options
+      Subserver.options
+    end
+
+    def boot_system
+      ENV['RACK_ENV'] = ENV['RAILS_ENV'] = environment
+
+      raise ArgumentError, "#{options[:require]} does not exist" unless File.exist?(options[:require])
+
+      if File.directory?(options[:require])
+        require 'rails'
+        if ::Rails::VERSION::MAJOR < 4
+          raise "Subserver does not support this version of Rails."
+        elsif ::Rails::VERSION::MAJOR == 4
+          require File.expand_path("#{options[:require]}/config/application.rb")
+          ::Rails::Application.initializer "subserver.eager_load" do
+            ::Rails.application.config.eager_load = true
+          end
+          require 'subserver/rails'
+          require File.expand_path("#{options[:require]}/config/environment.rb")
+        else
+          require 'subserver/rails'
+          require File.expand_path("#{options[:require]}/config/environment.rb")
+        end
+        options[:tag] ||= default_tag
+      else
+        not_required_message = "#{options[:require]} was not required, you should use an explicit path: " +
+            "./#{options[:require]} or /path/to/#{options[:require]}"
+
+        require(options[:require]) || raise(ArgumentError, not_required_message)
+      end
+    end
+
+    def default_tag
+      dir = ::Rails.root
+      name = File.basename(dir)
+      if name.to_i != 0 && prevdir = File.dirname(dir) # Capistrano release directory?
+        if File.basename(prevdir) == 'releases'
+          return File.basename(File.dirname(prevdir))
+        end
+      end
+      name
+    end
+
+    def validate!
+      options[:queues] << 'default' if options[:queues].empty?
+
+      if !File.exist?(options[:require]) ||
+         (File.directory?(options[:require]) && !File.exist?("#{options[:require]}/config/application.rb"))
+        logger.info "=================================================================="
+        logger.info "  Please point subserver to a Rails 4/5 application or a Ruby file  "
+        logger.info "  to load your subscriber classes with -r [DIR|FILE]."
+        logger.info "=================================================================="
+        logger.info @parser
+        die(1)
+      end
+
+      raise ArgumentError, "#{timeout}: #{options[:timeout]} is not a valid value" if options.has_key?(:timeout) && options[:timeout].to_i <= 0
+    end
+
+    def parse_options(argv)
+      opts = {}
+
+      @parser = OptionParser.new do |o|
+        o.on "-c", "--credentials PATH", "Path to Google Cloud credentials JSON file." do |arg|
+          opts[:credentials] = arg
+        end
+
+        o.on '-d', '--daemon', "Daemonize process" do |arg|
+          opts[:daemon] = arg
+        end
+
+        o.on '-e', '--environment ENV', "Application environment" do |arg|
+          opts[:environment] = arg
+        end
+
+        o.on '-g', '--tag TAG', "Process tag for procline" do |arg|
+          opts[:tag] = arg
+        end
+
+        o.on '-p', '--project ID', "Google Cloud Project ID" do |arg|
+          opts[:project_id] = arg
+        end
+
+        o.on "-q", "--queue QUEUE", "Subscriber queues to process with this server" do |arg|
+          queue = arg
+          opts = (opts[:queues] ||= []) << queue
+        end
+
+        o.on '-r', '--require [PATH|DIR]', "Location of Rails application with subscribers or file to require" do |arg|
+          opts[:require] = arg
+        end
+
+        o.on '-t', '--timeout NUM', "Shutdown timeout" do |arg|
+          opts[:timeout] = Integer(arg)
+        end
+
+        o.on "-v", "--verbose", "Print more verbose output" do |arg|
+          opts[:verbose] = arg
+        end
+
+        o.on '-C', '--config PATH', "path to YAML config file" do |arg|
+          opts[:config_file] = arg
+        end
+
+        o.on '-L', '--logfile PATH', "path to writable logfile" do |arg|
+          opts[:logfile] = arg
+        end
+
+        o.on '-P', '--pidfile PATH', "path to pidfile" do |arg|
+          opts[:pidfile] = arg
+        end
+
+        o.on '-V', '--version', "Print version and exit" do |arg|
+          puts "Subserver #{Subserver::VERSION}"
+          die(0)
+        end
+      end
+
+      @parser.banner = "subserver [options]"
+      @parser.on_tail "-h", "--help", "Show help" do
+        logger.info @parser
+        die 1
+      end
+      @parser.parse!(argv)
+
+      opts
+    end
+
+    def initialize_logger
+      Subserver::Logging.initialize_logger(options[:logfile]) if options[:logfile]
+      Subserver.logger.level = ::Logger::DEBUG if options[:verbose]
+    end
+
+    def write_pid
+      if path = options[:pidfile]
+        pidfile = File.expand_path(path)
+        File.open(pidfile, 'w') do |f|
+          f.puts ::Process.pid
+        end
+      end
+    end
+
+  end
+end